The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science
Open accessAbstract
Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the “intellectual distance” between the knowledge embodied in research proposals and an evaluator’s own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assignment of evaluators and proposals to generate 2,130 evaluator–proposal pairs. We find that evaluators systematically give lower scores to research proposals that are closer to their own areas of expertise and to those that are highly novel. The patterns are consistent with biases associated with boundedly rational evaluation of new ideas. The patterns are inconsistent with intellectual distance simply contributing “noise” or being associated with private interests of evaluators. We discuss implications for policy, managerial intervention, and allocation of resources in the ongoing accumulation of scientific knowledge. This paper was accepted by Lee Fleming, entrepreneurship and innovation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Exploration and Exploitation in Organizational Learning
March, J. G. (1991) · Organization Science · doi:10.1287/orsc.2.1.71 · Cited by 21,942
Endogenous Technological Change
Romer, P. M. (1990) · Journal of Political Economy · doi:10.1086/261725 · Cited by 17,711
Endogenous Technological Change
Romer, P. (1989) · National Bureau of Economic Research · doi:10.3386/w3210 · Cited by 13,303
Judgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
Statistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
The New production of knowledge: the dynamics of science and research in contemporary societies
(1995) · Choice Reviews Online · doi:10.5860/choice.32-4463 · Cited by 7,046
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720 · cited 2×
“Contrary to the popular notion of a “marketplace for ideas,” in which the best ideas simply rise to the top, resource allocation in academic science is shaped in important ways by supporting institutions and processes (Kuhn 1962, Merton 1968, Dasgupta and David 1994, Stephan 2012).”
Technological paradigms and technological trajectories
Dosi, G. (1982) · Research Policy · doi:10.1016/0048-7333(82)90016-6 · Cited by 6,685
Newell, A. (1972) · Cited by 6,222
Judgment Under Uncertainty: Heuristics and Biases.
Shafer, G. R., Kahnerman, D., Slovic, P., & Tversky, A. (1984) · Journal of the American Statistical Association · doi:10.2307/2288362 · Cited by 5,657
(1968) · PubMed · Cited by 4,803
Judgment under uncertainty: List of contributors
Kahneman, D. J., Slovic, P., & Tversky, A. (1982) · Cited by 4,670
Judgment under uncertainty: Causality and attribution
Kahneman, D. J., Slovic, P., & Tversky, A. (1982) · Cited by 4,670
Chase, W. G., & Simon, H. A. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90004-2 · Cited by 4,479 · cited 1×
“…are even often seemingly automatic, and even instantaneous, as a result of knowledge stored and comprehended in “chunks” and mental maps of hierarchies, relationships, contingencies, and “configural rules” (Fitts and Posner 1967, Newell and Simon 1972, Chase and Simon 1973, Ericsson and Smith 1991).”
The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies.
Baber, Z., Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1995) · Contemporary Sociology · doi:10.2307/2076669 · Cited by 4,406
The Increasing Dominance of Teams in Production of Knowledge
Wuchty, S., Jones, B. F., & Uzzi, B. (2007) · Science · doi:10.1126/science.1136099 · Cited by 3,615
A dynamic model of process and product innovation
Utterback, J. M., & Abernathy, W. J. (1975) · Omega · doi:10.1016/0305-0483(75)90068-7 · Cited by 3,509
Recombinant Uncertainty in Technological Search
Fleming, L. (2001) · Management Science · doi:10.1287/mnsc.47.1.117.10671 · Cited by 2,873
Toward a new economics of science
Partha, D., & David, P. A. (1994) · Research Policy · doi:10.1016/0048-7333(94)01002-1 · Cited by 2,448
Induction: Processes of Inference, Learning, and Discovery
Holland, J. H., Holyoak, K. J., Nisbett, R. E., Thagard, P. R., & Smoliar, S. W. (1987) · IEEE Expert · doi:10.1109/mex.1987.4307100 · Cited by 2,305
MANSO, G. (2011) · The Journal of Finance · doi:10.1111/j.1540-6261.2011.01688.x · Cited by 1,955
(1991) · Cited by 1,767
Induction: Processes Of Inference
Holland, J. H., Holyoak, K. J., Nisbett, R. E., & Thagard, P. (1986) · The MIT Press eBooks · doi:10.7551/mitpress/3729.001.0001 · Cited by 1,534
Atypical Combinations and Scientific Impact
Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013) · Science · doi:10.1126/science.1240474 · Cited by 1,457
Ericsson, K. A. (1996) · Cited by 1,285
The explicit economics of knowledge codification and tacitness
Cowan, R. (2000) · Industrial and Corporate Change · doi:10.1093/icc/9.2.211 · Cited by 1,274
How Professors Think: Inside the Curious World of Academic Judgment
Lamont, M. (2009) · doi:10.4159/9780674054158 · Cited by 1,086
The Burden of Knowledge and the “Death of the Renaissance Man”: Is Innovation Getting Harder?
JONES, B. F. (2009) · Review of Economic Studies · doi:10.1111/j.1467-937x.2008.00531.x · Cited by 1,058
Ambiguity Aversion and Comparative Ignorance
Fox, C. R., & Tversky, A. (1995) · The Quarterly Journal of Economics · doi:10.2307/2946693 · Cited by 1,022
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Weitzman, M. L. (1998) · The Quarterly Journal of Economics · doi:10.1162/003355398555595 · Cited by 979
Stephan, P. (2012) · Harvard University Press eBooks · doi:10.4159/harvard.9780674062757 · Cited by 934
Product Familiarity and Learning New Information
Johnson, E. J., & Russo, J. E. (1984) · Journal of Consumer Research · doi:10.1086/208990 · Cited by 922
Stephan, P. E. (2010) · Handbook of the Economics of Innovation · doi:10.1016/s0169-7218(10)01005-1 · Cited by 906
Johnson, E. J., & Payne, J. W. (1985) · Management Science · doi:10.1287/mnsc.31.4.395 · Cited by 829
The double‐edged sword of recombination in breakthrough innovation
Kaplan, S., & Vakili, K. (2015) · Strategic Management Journal · doi:10.1002/smj.2294 · Cited by 776
Two Approaches to the Study of Experts' Characteristics
Chi, M. T. H. (2006) · The Cambridge Handbook of Expertise and Expert Performance · doi:10.1017/cbo9780511816796.002 · Cited by 741
Ambiguity Aversion and Comparative Ignorance
Fox, C. R., & Tversky, A. (2000) · Choices, Values, and Frames · doi:10.1017/cbo9780511803475.031 · Cited by 670
Chance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · cited 3×
“Variance among reviewers is sometimes greater than variance between submissions (Cole et al. 1981).”
BackgroundHow professors think: inside the curious world of academic judgment
(2009) · Choice Reviews Online · doi:10.5860/choice.47-1492 · Cited by 601
Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts
Mollick, E., & Nanda, R. (2016) · Management Science · doi:10.1287/mnsc.2015.2207 · Cited by 515
Intellectual Property Rights and Innovation: Evidence from the Human Genome
Williams, H. L. (2013) · Journal of Political Economy · doi:10.1086/669706 · Cited by 506
Piezunka, H., & Dahlander, L. (2015) · Academy of Management Journal · doi:10.5465/amj.2012.0458 · Cited by 497
In one word: Not from experience
Brehmer, B. (1980) · Acta Psychologica · doi:10.1016/0001-6918(80)90034-7 · Cited by 465
Rhee, M., & Kim, T. (2016) · The Palgrave Encyclopedia of Strategic Management · doi:10.1057/978-1-349-94848-2_388-1 · Cited by 435
Technological guideposts and innovation avenues
Sahal, D. (1993) · Research Policy · doi:10.1016/0048-7333(93)90058-p · Cited by 423
Hallen, B. L. (2008) · Administrative Science Quarterly · doi:10.2189/asqu.53.4.685 · Cited by 418
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Technological guideposts and innovation avenues
Sahal, D. (1985) · Research Policy · doi:10.1016/0048-7333(85)90015-0 · Cited by 385
Climbing atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research
Furman, J. L., & Stern, S. (2011) · American Economic Review · doi:10.1257/aer.101.5.1933 · Cited by 344
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kühn, T., & Hacking, I. (2012) · Cited by 343
(1991) · CaltechAUTHORS (California Institute of Technology) · Cited by 340
3,000 Raw Ideas = 1 Commercial Success!
Stevens, G. A., & Burley, J. (1997) · Research-Technology Management · doi:10.1080/08956308.1997.11671126 · Cited by 339
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008) · American Psychologist · doi:10.1037/0003-066x.63.3.160 · Cited by 299
Exploring the Foundations of Cumulative Innovation: Implications for Organization Science
Murray, F., & O'Mahony, S. (2007) · Organization Science · doi:10.1287/orsc.1070.0325 · Cited by 278
The Effects of Blinding on the Quality of Peer Review: A Randomized Trial
McNutt, R. A. (1990) · JAMA · doi:10.1001/jama.1990.03440100079012 · Cited by 261
Rothwell, P. M. (2000) · Brain · doi:10.1093/brain/123.9.1964 · Cited by 253
New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System
Travis, G. D. L., & Collins, H. M. (1991) · Science, Technology, & Human Values · doi:10.1177/016224399101600303 · Cited by 245
Medical Subject Headings Used to Search the Biomedical Literature
Coletti, M. H., & Bleich, H. L. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080317 · Cited by 236
The effects of blinding on the quality of peer review. A randomized trial
Mcnutt, R. (1990) · JAMA · doi:10.1001/jama.263.10.1371 · Cited by 216
Expertise in auditing: Myth or reality?
Bédard, J. (1989) · Accounting, Organizations and Society · doi:10.1016/0361-3682(89)90037-8 · Cited by 194
Peerless Science: Peer Review and U.S. Science Policy.
Cole, S., Chubin, D. E., & Hackett, E. J. (1991) · Contemporary Sociology · doi:10.2307/2071849 · Cited by 189
Langfeldt, L. (2006) · Research Evaluation · doi:10.3152/147154406781776039 · Cited by 179
Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010072 · Cited by 178
Persistent nepotism in peer-review
Sandström, U., & Hällsten, M. (2008) · Scientometrics · doi:10.1007/s11192-008-0211-3 · Cited by 178
Expertise versus Bias in Evaluation: Evidence from the NIH
Li, D. (2017) · American Economic Journal: Applied Economics · doi:10.1257/app.20150421 · Cited by 169
Åstebro, T. B., & Elhedhli, S. (2006) · Management Science · doi:10.1287/mnsc.1050.0468 · Cited by 165
Graves, N., Barnett, A. G., & Clarke, P. (2011) · BMJ · doi:10.1136/bmj.d4797 · Cited by 140
Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2003) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/1467-985x.00278 · Cited by 139
Boudreau, K. J., & Lakhani, K. R. (2014) · Research Policy · doi:10.1016/j.respol.2014.08.001 · Cited by 135
Self-citationLee, J. (2015) · Journal of Computer-Mediated Communication · doi:10.1111/jcc4.12113 · Cited by 133
The acquisition of robust and flexible cognitive skills.
Taatgen, N. A., Huss, D., Dickison, D., & Anderson, J. R. (2008) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.137.3.548 · Cited by 115
The effects of blinding on the quality of peer review. A randomized trial.
McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990) · PubMed · Cited by 113
Is American Science in Decline?
Xie, Y., & Killewald, A. A. (2012) · Harvard University Press eBooks · doi:10.4159/harvard.9780674065048 · Cited by 110
The Validity of Peer Review in a General Medicine Journal
Jackson, J. L., Srinivasan, M., Rea, J., Fletcher, K. E., & Kravitz, R. L. (2011) · PLoS ONE · doi:10.1371/journal.pone.0022475 · Cited by 76
Bornmann, L., & Daniel, H. (2008) · Angewandte Chemie International Edition · doi:10.1002/anie.200800513 · Cited by 69
Multimethod study of clinical judgment
Johnson, P. E., Hassebrock, F., Durán, A. S., & Moller, J. H. (1982) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(82)90218-5 · Cited by 69
Campanario, J. M., & Acedo, E. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20556 · Cited by 62
An Examination of Sources of Peer-Review Bias
Blackburn, J. L., & Hakel, M. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01715.x · Cited by 56
A Kuhnian Critique of Psychometric Research on Peer Review
Lee, C. J. (2012) · Philosophy of Science · doi:10.1086/667841 · Cited by 48
Citation gamesmanship: testing for evidence of ego bias in peer review
Sugimoto, C. R., & Cronin, B. (2012) · Scientometrics · doi:10.1007/s11192-012-0845-z · Cited by 42
Cited by
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497
Enhancing the Effectiveness of Team Science
(2015) · National Academies Press eBooks · doi:10.17226/19007 · Cited by 620
Social Media and Their Affordances for Organizing: A Review and Agenda for Research
Leonardi, P. M., & Vaast, E. (2017) · Academy of Management Annals · doi:10.5465/annals.2015.0144 · Cited by 547
Bias against novelty in science: A cautionary tale for users of bibliometric indicators
Wang, J., Veugelers, R., & Stephan, P. (2017) · Research Policy · doi:10.1016/j.respol.2017.06.006 · Cited by 540
Data Analytics, Innovation, and Firm Productivity
Wu, L., Hitt, L., & Lou, B. (2020) · Management Science · doi:10.1287/mnsc.2018.3281 · Cited by 368
Metadata
- Authors
- Boudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C.
- Venue
- Management Science
- Published
- 2016-09
- Volume
- 62
- Issue
- 10
- Pages
- 2765-2783
- ISSN
- 0025-1909 (ISSN-L)
- 1526-5501
- OpenAlex
- W3122634626
- Cited by
- 481
- References
- 83
- FWCI
- 68.82
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.