The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index.
Accurately computing the log-sum-exp and softmax functions
Open accessAbstract
Abstract Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in low-precision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floating-point arithmetic and shifting can introduce subtractive cancellation. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones, so can safely be used, but that a division-free variant of softmax can suffer from loss of accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The softmax itself, in the standard max-shifted form, is backward stable with a condition number bounded by one in the relevant norm [34], so the delicacy we analyse is a property of A , not of the softmax evaluation.”
From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [34]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
(2016) · Cited by 11,679
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Accuracy and Stability of Numerical Algorithms: Second Edition
Higham, N. J. (2002) · doi:10.1137/1.9780898718027 · Cited by 3,803
Functions of Matrices: Theory and Computation
Higham, N. J. (2008) · doi:10.1137/1.9780898717778 · Cited by 2,113
Self-citationAccuracy and stability of numerical algorithms
(1996) · Choice Reviews Online · doi:10.5860/choice.34-0981 · Cited by 2,072
IEEE Standard for Floating-Point Arithmetic
Zuras, D., Cowlishaw, M., Aiken, A., et al. (2008) · doi:10.1109/ieeestd.2008.4610935 · Cited by 1,329
Computer Age Statistical Inference
Efron, B., & Hastie, T. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316576533 · Cited by 926
Research works printed but not linked: 7
- Goodfellow, 2016, Deep Learning
- Hewlett-Packard, 1982, HP-15C Advanced Functions Handbook. Portable Computer Division
- Higham, 2020, MIMS EPrint 2020.4
- Intel Corporation, 2018, BFLOAT16—Hardware Numerics Definition
- Jones, 2001, SciPy: Open Source Scientific Tools for Python.
- LeCun, The MNIST Database of Handwritten Digits.
- Murphy, 2012, Machine Learning: A Probabilistic Approach
Cited by
Sorted by citations
Items 101–127 of 127
Instantaneous Reproduction Number Estimation From Modelled IncidencePreprint
Challen, R., & Danon, L. (2025) · medRxiv · doi:10.64898/2025.12.11.25342071
METHODICAL INVESTIGATION OF LOSS FUNCTIONS IN DEEP LEARNING AND MACHINE LEARNING
Anbalagan, T., & Nath, M. K. (2025) · Biomedical Engineering Applications Basis and Communications · doi:10.4015/s101623722530007x
On the modulus of Lipschitz smoothness of log-sum-exp function
Luo, H., Wang, X., Werenski, M., & Yang, X. (2026) · Optimization Letters · doi:10.1007/s11590-026-02279-9
Bus dwell time estimation based solely on passenger tag-in and tag-out stamps from smart card dataPreprint
Jun, H., Kang, Y., & Sohn, K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6368103
RIS-aided Communication-Compatible MIMO Radar Unimodular Waveform Design
Zhong, K., Tai, X., Zhang, Y., Zuo, Y., Hu, J., Pan, C., & Li, H. (2025) · Globecom · doi:10.1109/globecom59602.2025.11432026
Entropy-Controlled Ant Systems with Strong Priors and Confidence Alignment
Jangra, R., Ahmed, B., Rampal, S., & Jakhar, Y. K. (2026) · International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC) · doi:10.1109/parc68365.2026.11453694
Deferred cyclotomic representation for stable and exact evaluation of q-hypergeometric series
Asante, S. K. (2026) · SciPost Physics Core · doi:10.21468/scipostphyscore.9.3.059
Zhou, Z., Wu, Q., & Wang, X. (2026) · Photonics · doi:10.3390/photonics13050405
Bai, S. (2026) · Research Square · doi:10.21203/rs.3.rs-8373845/v1
Dromigny, S., Zhao, X., Reu, T., Koelemeijer, P., & Curtis, A. (2026) · Geophysical Journal International · doi:10.1093/gji/ggag180
Kim, S., Cho, H., Jeong, H., Yoo, D., Lee, K., Lee, H. S., & Min, K. (2026) · One Health · doi:10.1016/j.onehlt.2026.101457
Solving hypervolume scalarizations for MOCO problems
Lopes, G., Klamroth, K., & Paquete, L. (2026) · Computers & Operations Research · doi:10.1016/j.cor.2026.107578
Probing the thermodynamic and kinetic sensitivities in ammonia synthesis
Alsaedi, S. M., Ward, K., Ross, A., & Dupont, V. A. (2026) · Next Materials · doi:10.1016/j.nxmate.2026.102468
Kim, H., Yoo, I., Kim, Y. L., Kwak, D.-R., Park, I. K., & Kang, D. (2026) · Nondestructive Testing And Evaluation · doi:10.1080/10589759.2026.2695279
Kemel, A., & Spada, L. L. (2026) · IEEE International Conference on Smart Information Systems and Technologies (SIST) · doi:10.1109/sist61674.2026.11596488
Exposure-Based Reinforcement Learning to Rank
Oosterhuis, H., Jagerman, R., Qin, Z., & Wang, X. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820396
Solanes, J. E., & Francés-Falip, A. (2026) · Electronics · doi:10.3390/electronics15143192
Bai, J., Fan, J., & Li, H. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103945
The Hidden Fragility of GNNs: How Graph Structure Amplifies Numerical Errors
Gu, J., & Qiao, Z. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818182
Situation-Aware Feedback-Predictive Control Framework for Lane-Less Dense Traffic
Khound, P., Bhave, N. J., Xie, S., & Chakraborty, D. (2026) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit70676.2026.11630772
Effective localization and forward stability of multinode Shepard operators
Dell’Accio, F., Tommaso, F. D., & Larosa, F. (2026) · Mathematics and Computers in Simulation · doi:10.1016/j.matcom.2026.08.003
Xiaolong, M., He, Y., Pan, S., Baoliang, W., Can, H., Zhihui, T., Bux, T. A., & Abdelhamid, M. A. (2026) · Smart Agricultural Technology · doi:10.1016/j.atech.2026.102505
Alnusayri, M., Mumtaz, S., Aldughayfiq, B., Allahem, H., Almashfi, N., AlHammadi, D. A., & Jalal, A. (2026) · Bioengineering · doi:10.3390/bioengineering13091082
Overhead-Aware Online Codebook Learning for STAR-RIS-Aided Near-Field MIMO Systems
Shu, Y.-E., Chang, R. Y., & Fang, S. (2026) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2026.3738827
Shared Wavelength Selection for Multicomponent Analysis of Corn by Near-Infrared Spectroscopy
Yang, X., Dong, S., Liu, X., & Chen, C. (2026) · Chemosensors · doi:10.3390/chemosensors14100234
Piqué, F., Bianchi, M., & Artoni, A. (2026) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2026.113635
Metadata
- Authors
- Blanchard, P., Higham, D. J., & Higham, N. J.
- Affiliations
- Department of Mathematics, University of Manchester, Manchester, M13 9PL, UK
- University of Edinburgh (GB)
- University of Manchester (GB)
- Corresponding author
- Higham, Nicholas J
- Venue
- IMA Journal of Numerical Analysis
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-08-19
- Volume
- 41
- Issue
- 4
- Pages
- 2311-2330
- ISSN
- 0272-4979 (ISSN-L)
- 1464-3642
- OpenAlex
- W3004059350
- Cited by
- 126
- References
- 31
- Citations by year
- 2026: 32 · 2025: 30 · 2024: 23 · 2023: 20 · 2022: 12 · 2021: 9
- FWCI
- 5.10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IMA Journal of Numerical Analysis (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion)
- MIMS EPrints (University of Southampton) (acceptedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/P020720/1)
- MathWorks
- Royal Society
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
- Physical Sciences › Mathematics › Modeling and Simulation › Mathematical and Computational Methods
- Physical Sciences › Mathematics › Applied Mathematics › Mathematical functions and polynomials
- Keywords
- softmax, floating-point arithmetic, rounding error analysis, condition number, overflow and underflow, numerical accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
