The Neural Hype and Comparisons Against Weak Baselines
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Recently, the machine learning community paused in a moment of self-reflection. In a widelydiscussed paper at ICLR 2018, Sculley et al. [13] wrote: "We observe that the rate of empirical advancement may not have been matched by consistent increase in the level of empirical rigor across the field as a whole." Their primary complaint is the development of a "research and publication culture that emphasizes wins" (emphasis in original), which typically means "demonstrating that a new method beats previous methods on a given task or benchmark". An apt description might be "leaderboard chasing"-and for many vision and NLP tasks, this isn't a metaphor. There are literally centralized leaderboards1 that track incremental progress, down to the fifth decimal point, some persisting over years, accumulating dozens of entries. Sculley et al. remind us that "the goal of science is not wins, but knowledge". The structure of the scientific enterprise today (pressure to publish, pace of progress, etc.) means that "winning" and "doing good science" are often not fully aligned. To wit, they cite a number of papers showing that recent advances in neural networks could very well be attributed to mundane issues like better hyperparameter optimization. Many results can't be reproduced, and some observed improvements might just be noise.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“In a seminal essay, Ioannidis [7] suggested that “the hotter a scientific field. . . the less likely the research findings are to be true” and that “claimed research findings may often be simply accurate measures of the prevailing bias”.”
Background
Cited by
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771
“Language Models While unsupervised neural retrieval is notoriously difficult to improve over traditional IR (Lin, 2019), we include them as baselines for comparison.”
BackgroundNogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615
“This, at least in part, prevented the neural ranking models from being successful when compared to more classical IR techniques (Lin, 2019).”
ResultAre we really making much progress? A worrying analysis of recent neural recommendation approaches
Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019) · Proceedings of the 13th ACM Conference on Recommender Systems · doi:10.1145/3298689.3347058 · Cited by 431
End-to-End Open-Domain Question Answering with
Yang, W., Xie, Y., Lin, A., Li, X., Tan, L., Xiong, K., Li, M., & Lin, J. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4013 · Cited by 391
Self-citationRecommender Systems Leveraging Multimedia Content
Deldjoo, Y., Schedl, M., Cremonesi, P., & Pasi, G. (2021) · ACM Computing Surveys · doi:10.1145/3407190 · Cited by 258
Metadata
- Authors
- Lin, J.
- Venue
- ACM SIGIR Forum
- Published
- 2019-01-17
- Volume
- 52
- Issue
- 2
- Pages
- 40-51
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W2910577570
- Cited by
- 140
- References
- 2
- FWCI
- 12.20
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.