The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index.
Accurately computing the log-sum-exp and softmax functions
Open accessAbstract
Abstract Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in low-precision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floating-point arithmetic and shifting can introduce subtractive cancellation. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones, so can safely be used, but that a division-free variant of softmax can suffer from loss of accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The softmax itself, in the standard max-shifted form, is backward stable with a condition number bounded by one in the relevant norm [34], so the delicacy we analyse is a property of A , not of the softmax evaluation.”
From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [34]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
(2016) · Cited by 11,679
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Accuracy and Stability of Numerical Algorithms: Second Edition
Higham, N. J. (2002) · doi:10.1137/1.9780898718027 · Cited by 3,803
Functions of Matrices: Theory and Computation
Higham, N. J. (2008) · doi:10.1137/1.9780898717778 · Cited by 2,113
Self-citationAccuracy and stability of numerical algorithms
(1996) · Choice Reviews Online · doi:10.5860/choice.34-0981 · Cited by 2,072
IEEE Standard for Floating-Point Arithmetic
Zuras, D., Cowlishaw, M., Aiken, A., et al. (2008) · doi:10.1109/ieeestd.2008.4610935 · Cited by 1,329
Computer Age Statistical Inference
Efron, B., & Hastie, T. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316576533 · Cited by 926
Bayesian classification with Gaussian processes
Williams, C. K. I., & Barber, D. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.735807 · Cited by 764
Elementary functions: algorithms and implementation
(1998) · Choice Reviews Online · doi:10.5860/choice.35-2759 · Cited by 531
Handbook of Floating-Point Arithmetic
Muller, J., Brisebarre, N., de Dinechin, F. D., Jeannerod, C.-P., Lefèvre, V., Melquiond, G., Revol, N., Stehlé, D., & Torres, S. (2009) · Birkhäuser Boston eBooks · doi:10.1007/978-0-8176-4705-6 · Cited by 460
The Accuracy of Floating Point Summation
Higham, N. J. (1993) · SIAM Journal on Scientific Computing · doi:10.1137/0914050 · Cited by 268
Computer age statistical inference algorithms, evidence, and data science
Efron, B., & Hastie, T. J. (2016) · Thammasat University Digital Collections · Cited by 262
Handbook of Floating-Point Arithmetic
Muller, J., Brunie, N., de Dinechin, F. D., Jeannerod, C.-P., Joldeş, M., Lefèvre, V., Melquiond, G., Revol, N., & Torres, S. (2018) · Springer eBooks · doi:10.1007/978-3-319-76526-6 · Cited by 234
A High-Speed and Low-Complexity Architecture for Softmax Function in Deep Learning
Wang, M., Lu, S., Zhu, D., Lin, J. T., & Wang, Z. (2018) · IEEE Asia Pacific Conference on Circuits and Systems (APCCAS) · doi:10.1109/apccas.2018.8605654 · Cited by 225
Muller, J. (2016) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4899-7983-4 · Cited by 103
A New Approach to Probabilistic Rounding Error Analysis
Higham, N. J., & Mary, T. (2019) · SIAM Journal on Scientific Computing · doi:10.1137/18m1226312 · Cited by 94
Simulating Low Precision Floating-Point Arithmetic
Higham, N. J., & Pranesh, S. (2019) · SIAM Journal on Scientific Computing · doi:10.1137/19m1251308 · Cited by 73
Log-Sum-Exp Neural Networks and Posynomial Models for Convex and Log-Log-Convex Data
Calafiore, G. C., Gaubert, S., & Possieri, C. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2910417 · Cited by 69
Error estimation of floating-point summation and dot product
Rump, S. M. (2011) · BIT Numerical Mathematics · doi:10.1007/s10543-011-0342-4 · Cited by 46
A Class of Fast and Accurate Summation Algorithms
Pierre, B., Higham, N. J., & Mary, T. (2020) · SIAM Journal on Scientific Computing · doi:10.1137/19m1257780 · Cited by 43
Fast and correctly rounded logarithms in double-precision
de Dinechin, F., Lauter, C., & Muller, J. (2007) · RAIRO - Theoretical Informatics and Applications · doi:10.1051/ita:2007003 · Cited by 41
The Matrix Unwinding Function, with an Application to Computing the Matrix Exponential
Aprahamian, M., & Higham, N. J. (2014) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/130920137 · Cited by 20
Secure multiparty computations in floating-point arithmetic
Guo, C., Hannun, A., Knott, B., van der Maaten, L., Tygert, M., & Zhu, R. (2020) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaaa038 · Cited by 18
Sharp estimates for perturbation errors in summations
Lange, M., & Rump, S. M. (2018) · Mathematics of Computation · doi:10.1090/mcom/3355 · Cited by 14
Secure multiparty computations in floating-point arithmeticPreprint
Guo, C., Hannun, A., Knott, B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03192 · Cited by 11
Softmax Optimizations for Intel Xeon Processor-based PlatformsPreprint
Czaja, J., Gallus, M., Patejko, T., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.12380 · Cited by 3
Sparse ℓ1- and ℓ2-Center Classifiers
Calafiore, G. C., & Fracastoro, G. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3036838 · Cited by 2
Research works printed but not linked: 7
- Goodfellow, 2016, Deep Learning
- Hewlett-Packard, 1982, HP-15C Advanced Functions Handbook. Portable Computer Division
- Higham, 2020, MIMS EPrint 2020.4
- Intel Corporation, 2018, BFLOAT16—Hardware Numerics Definition
- Jones, 2001, SciPy: Open Source Scientific Tools for Python.
- LeCun, The MNIST Database of Handwritten Digits.
- Murphy, 2012, Machine Learning: A Probabilistic Approach
Cited by
Sorted by citations
Goal-driven Self-Attentive Recurrent Networks for Trajectory Prediction
Chiara, L. F., Coscia, P., Das, S., Calderara, S., Cucchiara, R., & Ballan, L. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00282 · Cited by 80
Wang, J., Yang, Z., Chen, C., Ge, Y., Wan, X., Bao, S., Ding, J., Wang, L., & Jiang, H. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae387 · Cited by 63
Bemporad, A. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3183036 · Cited by 53
SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
Wang, H., Fang, J., Tang, X., Yue, Z., Li, J., Qin, Y., Guan, S., Yang, Q., Wang, Y., Li, C., et al. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00093 · Cited by 38
Fraley, C., Raftery, A. E., & Scrucca, L. (1999) · doi:10.32614/cran.package.mclust · Cited by 31
Metadata
- Authors
- Blanchard, P., Higham, D. J., & Higham, N. J.
- Affiliations
- Department of Mathematics, University of Manchester, Manchester, M13 9PL, UK
- University of Edinburgh (GB)
- University of Manchester (GB)
- Corresponding author
- Higham, Nicholas J
- Venue
- IMA Journal of Numerical Analysis
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-08-19
- Volume
- 41
- Issue
- 4
- Pages
- 2311-2330
- ISSN
- 0272-4979 (ISSN-L)
- 1464-3642
- OpenAlex
- W3004059350
- Cited by
- 126
- References
- 31
- Citations by year
- 2026: 32 · 2025: 30 · 2024: 23 · 2023: 20 · 2022: 12 · 2021: 9
- FWCI
- 5.10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IMA Journal of Numerical Analysis (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion)
- MIMS EPrints (University of Southampton) (acceptedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/P020720/1)
- MathWorks
- Royal Society
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Mathematics › Modeling and Simulation › Mathematical and Computational Methods
- Physical Sciences › Mathematics › Applied Mathematics › Mathematical functions and polynomials
- Keywords
- softmax, floating-point arithmetic, rounding error analysis, condition number, overflow and underflow, numerical accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
