CoChat
Sign Up
Cited by 113Meta-analysis and systematic reviewsscientometrics and bibliometrics researchAcademic Publishing and Open Access

Nonstandard Errors

Open access
MENKVELD, ALBERT J., DREBER, ANNA, HOLZMEISTER, FELIX, HUBER, JUERGEN, JOHANNESSON, MAGNUS, KIRCHLER, MICHAEL, NEUSÜß, SEBASTIAN, RAZEN, MICHAEL +335 more·2024·The Journal of Finance
DOI 10.1111/jofi.13337
Open this paper in CoChat — free account

Abstract

In statistics, samples are drawn from a population in a data‐generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence‐generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer‐review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/jofi.13337

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 26

  1. Angel, James J., Lawrence E. Harris, and Chester S. Spatt. 2015. 'Eq- uity Trading in the 21st Century: An Update.' Quarterly Journal of Finance 5:1-39.
  2. Bao, Jack, Maureen O'Hara, and Xing (Alex) Zhou. 2018. 'The Volcker Rule and Corporate Bond Market Making in Times of Stress.' Journal of Financial Economics 130:95-113.
  3. Blume, Marshall E. 1974. 'Unbiased Estimators of Long-Run Expected Rates of Return.' Journal of the American Statistical Association 69:634-638.
  4. Chen, Andrew Y. 2021. 'The Limits of p -Hacking: Some Thought Experiments.' Journal of Finance 76:2447-2480.
  5. Geyer-Klingeberg, Jerome, Markus Hang, and Andreas Rathgeber. 2020. 'Meta-analysis in finance research: Opportunities, challenges, and contemporary applications.' International Review of Financial Analysis 7:1-15.
  6. Harvey, Campbell R. 2017. 'Presidential Address: The Scientific Outlook in Financial Economics.' Journal of Finance 72:13991440.
  7. Harvey, Campbell R. and Yan Liu. 2020. 'False (and Missed) Discoveries in Financial Economics.' Journal of Finance 75:2503-2553.
  8. Hendershott, Terrence and Albert J. Menkveld. 2014. 'Price Pressures.' Journal of Financial Economics 114:405-423.
  9. Hou, Kewei, Chen Xue, and Lu Zhang. 2018. 'Replicating Anomalies.' Review of Financial Studies 33:2019-2133.
  10. Jensen, Theis Ingerslev, Bryan Kelly, and Lasse Pedersen. 2022. 'Is There a Replication Crisis in Finance.' Journal of Finance (forthcoming) .
  11. Jovanovic, Boyan and Albert J. Menkveld. 2021. 'Equilibrium BidPrice Dispersion.' Journal of Political Economy (forthcoming) .
  12. Kirchler, Michael, Florian Lindner, and Utz Weitzel. 2018. 'Rankings and Risk-Taking in the Finance Industry.' Journal of Finance 73:2271-2302.
  13. K¨ oszegi, Botond. 2006. 'Ego Utility, Overconfidence, and Task Choice.' Journal of the European Economic Association 4:673707.
  14. Leamer, Edward E. 1983. 'Let's Take the Con Out of Econometrics.' American Economic Review 73:31-43.
  15. Linnainmaa, Juhani T. and Michael R. Roberts. 2018. 'The History of the Cross-Section of Stock Returns.' Review of Financial Studies 31:2606-2649.
  16. Liu, Yang, Alex Kale, Tim Althoff, and Jeffrey Heer. 2021. 'Boba: Authoring and Visualizing Multiverse Analyses.' IEEE Transactions on Visualization and Computer Graphics 27:1753-1763.
  17. Mas-Colell, Andreu, Michael D. Whinston, and Jerry R. Green. 1995. Microeconomic Theory . New York: Oxford University Press.
  18. Mavroeidis, Sophocles, Mikkel Plagborg-Møller, and James H. Stock. 2014. 'Empirical Evidence on Inflation Expectations in the New Keynesian Phillips Curve.' Journal of Economic Literature 52:124-188.
  19. Menkveld, Albert J. 2016. 'The Economics of High-Frequency Trading: Taking Stock.' Annual Review of Financial Economics 8:1-24.
  20. Mitton, Todd. 2021. 'Methodological Variation in Empirical Corporate Finance.' Review of Financial Studies (forthcoming) .
  21. Parlour, Christine A. and Duane J. Seppi. 2008. 'Limit Order Markets: A Survey.' In Handbook of Financial Intermediation and Banking , edited by Arnoud W.A. Boot and Anjan V. Thakor. Amsterdam, Netherlands: Elsevier Publishing.
  22. Pastor, L. and R.F. Stambaugh. 2003. 'Liquidity Risk and Expected Returns.' Journal of Political Economy 111:642-685.
  23. Scholz, Fritz W. and Michael A. Stephens. 1987. 'K-Sample Anderson-Darling Tests.' Journal of the American Statistical Association 82:918-924.
  24. ˇ Sid´ ak, Zbynˇ e. 1967. 'Rectangular confidence regions for the means of multivariate normal distributions.' Journal of the American Statistical Association 62 (318):626-633.
  25. Tran, Anh and Richard Zeckhauser. 2012. 'Rank as an Inherent Incentive: Evidence from a Field Experiment.' Journal of Public Economics 96:645-650.
  26. Yule, G. Udny. 1897. 'On the Significance of Bravais' Formulae for Regression.' Proceedings of the Royal Society of London 60:477489.

Cited by

Sorted by citations

Authors
MENKVELD, A. J., DREBER, A., HOLZMEISTER, F., HUBER, J., JOHANNESSON, M., KIRCHLER, M., NEUSÜß, S., RAZEN, M., WEITZEL, U., ABAD‐DÍAZ, D., ABUDY, M. (., ADRIAN, T., AIT‐SAHALIA, Y., AKMANSOY, O., ALCOCK, J. T., ALEXEEV, V., ALOOSH, A., AMATO, L., AMAYA, D., ANGEL, J. J., AVETIKIAN, A. T., BACH, A., BAIDOO, E., BAKALLI, G., BAO, L., BARBON, A., BASHCHENKO, O., BINDRA, P. C., BJØNNES, G. H., BLACK, J. R., BLACK, B. S., BOGOEV, D., CORREA, S. B., BONDARENKO, O., BOS, C. S., BOSCH‐ROSA, C., BOURI, E., BROWNLEES, C., CALAMIA, A., CAO, V. N., CAPELLE‐BLANCARD, G., ROMERO, L. M. C., CAPORIN, M., CARRION, A., CASKURLU, T., CHAKRABARTY, B., CHEN, J., CHERNOV, M., CHEUNG, W., CHINCARINI, L. B., CHORDIA, T., CHOW, S., CLAPHAM, B., COLLIARD, J., COMERTON‐FORDE, C., CURRAN, E., DAO, T., DARE, W., DAVIES, R. J., BLASIS, R. D., NARD, G. F. D., DECLERCK, F., DEEV, O., DEGRYSE, H., DEKU, S. Y., DESAGRE, C., DIJK, M. A. V., DIM, C., DIMPFL, T., DONG, Y. J., DRUMMOND, P. A., DUDDA, T., DUEVSKI, T., DUMITRESCU, A., DYAKOV, T., DYHRBERG, A. H., DZIELIŃSKI, M., EKSI, A., KALAK, I. E., ELLEN, S. T., EUGSTER, N., EVANS, M. D. D., FARRELL, M., FELEZ‐VINAS, E., FERRARA, G., FERROUHI, E. M., FLORI, A., FLUHARTY‐JAIDEE, J. T., FOLEY, S. D. V., FONG, K. Y. L., FOUCAULT, T., FRANUS, T., FRANZONI, F., FRIJNS, B., FRÖMMEL, M., FU, S. M., FÜLLBRUNN, S. C., GAN, B., GAO, G., GEHRIG, T. P., GEMAYEL, R., GERRITSEN, D., GIL‐BAZO, J., GILDER, D., GLOSTEN, L. R., GOMEZ, T., GORBENKO, A., GRAMMIG, J., GRÉGOIRE, V., GÜÇBILMEZ, U., HAGSTRÖMER, B., HAMBUCKERS, J., HAPNES, E., HARRIS, J. H., HARRIS, L., HARTMANN, S., HASSE, J., HAUTSCH, N., HE, X. (., HEATH, D., HEDIGER, S., HENDERSHOTT, T., HIBBERT, A. M., HJALMARSSON, E., HOELSCHER, S. A., HOFFMANN, P., HOLDEN, C. W., HORENSTEIN, A. R., HUANG, W., HUANG, D., HURLIN, C., ILCZUK, K., IVASHCHENKO, A., IYER, S. R., JAHANSHAHLOO, H., JALKH, N., JONES, C. M., JURKATIS, S., JYLHÄ, P., KAECK, A. T., KAISER, G., KARAM, A., KARMAZIENE, E., KASSNER, B., KAUSTIA, M., KAZAK, E., KEARNEY, F., KERVEL, V. V., KHAN, S. A., KHOMYN, M. K., KLEIN, T., KLEIN, O., KLOS, A., KOETTER, M., KOLOKOLOV, A., KORAJCZYK, R. A., KOZHAN, R., KRAHNEN, J. P., KUHLE, P., KWAN, A., LAJAUNIE, Q., LAM, F. Y. E. C., LAMBERT, M., LANGLOIS, H., LAUSEN, J., LAUTER, T., LEIPPOLD, M., LEVIN, V., LI, Y., LI, H., LIEW, C. Y., LINDNER, T., LINTON, O., LIU, J., LIU, A., LLORENTE, G., LOF, M., LOHR, A., LONGSTAFF, F., LOPEZ‐LIRA, A., MANKAD, S., MANO, N., MARCHAL, A., MARTINEAU, C., MAZZOLA, F., MELOSO, D., MI, M. G., MIHET, R., MOHAN, V., MOINAS, S., MOORE, D., MU, L., MURAVYEV, D., MURPHY, D., NESZVEDA, G., NEUMEIER, C., NIELSSON, U., NIMALENDRAN, M., NOLTE, S., NORDEN, L. L., O'NEILL, P., OBAID, K., ØDEGAARD, B. A., ÖSTBERG, P., PAGNOTTA, E., PAINTER, M., PALAN, S., PALIT, I. J., PARK, A., PASCUAL, R., PASQUARIELLO, P., PASTOR, L., PATEL, V., PATTON, A. J., PEARSON, N. D., PELIZZON, L., PELLI, M., PELSTER, M., PÉRIGNON, C., PFIFFER, C., PHILIP, R., PLÍHAL, T., PRAKASH, P., PRESS, O., PRODROMOU, T., PROKOPCZUK, M., PUTNINS, T., QIAN, Y., RAIZADA, G., RAKOWSKI, D., RANALDO, A., REGIS, L., REITZ, S., RENAULT, T., RENJIE, R. W., RENO, R., RIDDIOUGH, S. J., RINNE, K., RINTAMÄKI, P., RIORDAN, R., RITTMANNSBERGER, T., LONGARELA, I. R., ROESCH, D., ROGNONE, L., ROSEMAN, B., ROŞU, I., ROY, S., RUDOLF, N., RUSH, S. R., RZAYEV, K., RZEŹNIK, A. A., SANFORD, A., SANKARAN, H., SARKAR, A., SARNO, L., SCAILLET, O., SCHARNOWSKI, S., SCHENK‐HOPPÉ, K. R., SCHERTLER, A., SCHNEIDER, M., SCHROEDER, F., SCHÜRHOFF, N., SCHUSTER, P., SCHWARZ, M. A., SEASHOLES, M. S., SEEGER, N. J., SHACHAR, O., SHKILKO, A., SHUI, J., SIKIC, M., SIMION, G., SMALES, L. A., SÖDERLIND, P., SOJLI, E., SOKOLOV, K., SÖNKSEN, J., SPOKEVICIUTE, L., STEFANOVA, D., SUBRAHMANYAM, M. G., SZASZI, B., TALAVERA, O., TANG, Y., TAYLOR, N., THAM, W. W., THEISSEN, E., THIMME, J., TONKS, I., TRAN, H., TRAPIN, L., TROLLE, A. B., VADUVA, M. A., VALENTE, G., NESS, R. A. V., VASQUEZ, A., VEROUSIS, T., VERWIJMEREN, P., VILHELMSSON, A., VILKOV, G., VLADIMIROV, V., VOGEL, S., VOIGT, S., WAGNER, W., WALTHER, T., WEISS, P., WEL, M. V. D., WERNER, I. M., WESTERHOLM, P. J., WESTHEIDE, C., WIKA, H. C., WIPPLINGER, E., WOLF, M., WOLFF, C. C. P., WOLK, L., WONG, W., WRAMPELMEYER, J., WU, Z., XIA, S., XIU, D., XU, K., XU, C., YADAV, P. K., YAGÜE, J., YAN, C., YANG, A., YOO, W., YU, W., YU, Y., YU, S., YUESHEN, B. Z., YUFEROVA, D., ZAMOJSKI, M., ZAREEI, A., ZEISBERGER, S. M., ZHANG, L., ZHANG, S. S., ZHANG, X., ZHAO, L., ZHONG, Z., ZHOU, Z. I., ZHOU, C., ZHU, X. S., ZOICAN, M., & ZWINKELS, R.
Affiliations
  • University of Amsterdam (NL)
  • Stockholm School of Economics (SE)
  • Universität Innsbruck (AT)
  • Aalto University (FI)
  • Vrije Universiteit Amsterdam (NL)
  • Barcelona School of Economics (ES)
  • Bar-Ilan University (IL)
  • International Monetary Fund (US)
  • Centre National de la Recherche Scientifique (FR)
  • European Central Bank (DE)
  • Wilfrid Laurier University (CA)
  • Georgetown University (US)
  • INSEAD (FR)
  • Bank of England (GB)
  • Auburn University (US)
  • Bank for International Settlements (CH)
  • HEC Paris (FR)
  • BI Norwegian Business School (NO)
  • EDF Energy (United Kingdom) (GB)
  • University of Liège (BE)
  • Norges Bank (NO)
  • Durham University (GB)
  • Bowling Green State University (US)
  • Eötvös Loránd University (HU)
  • Emory University (US)
  • Goethe University Frankfurt (DE)
  • KU Leuven (BE)
  • Columbia University (US)
  • Erasmus University Rotterdam (NL)
  • Frankfurt School of Finance & Management (DE)
  • Esade (ES)
  • Ecole des Hautes Etudes Commerciales du Nord (FR)
  • Université Ibn-Tofail (MA)
  • Ghent University (BE)
  • Federal Reserve Bank of New York (US)
  • École Polytechnique Fédérale de Lausanne (CH)
  • HEC Montréal (CA)
  • Aix-Marseille Sciences Economiques (FR)
  • Central Michigan University (US)
  • John von Neumann University (HU)
  • Karlsruhe Institute of Technology (DE)
  • Indian Institute of Management Ahmedabad (IN)
  • Christian-Albrechts-Universität zu Kiel (DE)
  • Halle Institute for Economic Research (DE)
  • Cornell University (US)
  • Hong Kong Institute for Monetary and Financial Research (HK)
  • Justus-Liebig-Universität Gießen (DE)
  • Duke University (US)
  • Federal Housing Finance Agency (US)
  • Instituto Tecnológico Autónomo de México (MX)
  • Deutsche Bundesbank (DE)
ORCID
Venue
The Journal of Finance
Publisher
Wiley
Published
2024-04-17
Volume
79
Issue
3
Pages
2339-2390
ISSN
  • 0022-1082 (ISSN-L)
  • 1540-6261
DOI
10.1111/jofi.13337
OpenAlex
W4394881644
Cited by
113
References
8
Citations by year
2026: 42 · 2025: 36 · 2024: 29 · 2023: 3 · 2022: 3
FWCI
29.87
Type
journal article
Language
en
Open access
Licence
Locations
  • The Journal of Finance (publishedVersion, cc-by)
  • Hispana (publishedVersion, cc-by)
  • Durham Research Online (Durham University) (publishedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • Research Explorer (The University of Manchester) (publishedVersion, cc-by)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (publishedVersion, cc-by)
  • WU Research (publishedVersion, cc-by)
  • City Research Online (City University London) (acceptedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by-nc-nd)
  • Toulouse 1 Capitole Publications (Université Toulouse I Capitole) (acceptedVersion)
  • HAL AMU (submittedVersion, cc-by-nd)
  • Ghent University Academic Bibliography (Ghent University) (submittedVersion, cc-by)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
  • Dublin City University Open Access Institutional Repository (Dublin City University) (submittedVersion)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion, cc-by-sa)
  • London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • IRIS - Institutional Research Information System (Libera Università Internazionale degli Studi Sociali Guido Carli) (submittedVersion, cc-by)
  • Institutional Research Information System University of Turin (University of Turin) (submittedVersion, other-oa)
  • ARCA (Università Ca' Foscari Venezia) (submittedVersion, cc-by)
  • Università Politecnica delle Marche (Università Politecnica delle Marche) (submittedVersion, cc-by)
  • Lirias (KU Leuven) (submittedVersion, cc-by)
  • Tuwhera (Auckland University of Technology) (submittedVersion, cc-by)
  • Research Portal (Queen's University Belfast) (submittedVersion, cc-by)
  • Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion, other-oa)
  • Open Access at Essex (University of Essex) (submittedVersion)
  • London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (submittedVersion)
  • RUA, Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) (submittedVersion, cc-by)
  • MADOC (University of Mannheim) (submittedVersion, cc-by)
  • UEA Digital Repository (University of East Anglia) (submittedVersion)
  • Warwick Research Archive Portal (University of Warwick) (submittedVersion)
  • Padua Research Archive (University of Padova) (submittedVersion, cc-by)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by-nc-nd)
  • Monash University Research Portal (Monash University) (submittedVersion, cc-by)
  • UWA Profiles and Research Repository (UWA) (submittedVersion, cc-by)
  • HAL (submittedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by-nc)
  • Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, other-oa)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion, other-oa)
  • Hispana (submittedVersion, cc-by)
  • IRIS (submittedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
  • Lund University Publications (Lund University) (submittedVersion)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, public-domain)
  • CBS Research Portal (Copenhagen Business School) (submittedVersion, cc-by)
  • Bristol Research (University of Bristol) (submittedVersion, cc-by)
  • DSpace (Open University in the Netherlands) (submittedVersion, cc-by)
  • SERVAL (Université de Lausanne) (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Knut och Alice Wallenbergs Stiftelse
  • Riksbankens Jubileumsfond (P21‐0168)
  • Austrian Science Fund (P29362, SFBF63)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
research reproducibility, peer review, scientific uncertainty, hypothesis testing, research quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review