Cited by 127Numerical Methods and AlgorithmsMathematical and Computational MethodsMathematical functions and polynomials

Accurately computing the log-sum-exp and softmax functions

Open access
Blanchard, Pierre, Higham, Desmond J, Higham, Nicholas J·2021·IMA Journal of Numerical Analysis

Abstract

Abstract Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in low-precision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floating-point arithmetic and shifting can introduce subtractive cancellation. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones, so can safely be used, but that a division-free variant of softmax can suffer from loss of accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/imanum/draa038

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The softmax itself, in the standard max-shifted form, is backward stable with a condition number bounded by one in the relevant norm, so the delicacy analysed is a property of A, not of the softmax evaluation.

    “The softmax itself, in the standard max-shifted form, is backward stable with a condition number bounded by one in the relevant norm [34], so the delicacy we analyse is a property of A , not of the softmax evaluation.”

    From Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention (2026) · cites this paper as [34]

    Section 2 (page unavailable) explicitly says the softmax condition number in the infinity norm can be arbitrarily large and that shifting does not change it, contrary to the claimed bound of one. Section 4, Theorem 4.3 (page unavailable), gives a forward-error bound for the max-shifted evaluation, not a backward-stability result. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 31 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Murphy, K. P. (2012) Machine Learning: A Probabilistic Approach. Cambridge, UK: Cambridge University Press.

Cited by

Sorted by citations

Authors
Blanchard, P., Higham, D. J., & Higham, N. J.
Affiliations
  • Department of Mathematics, University of Manchester, Manchester, M13 9PL, UK
  • University of Edinburgh (GB)
  • University of Manchester (GB)
Corresponding author
Higham, Nicholas J
Venue
IMA Journal of Numerical Analysis
Published
2020-08-19
Volume
41
Issue
4
Pages
2311-2330
ISSN
  • 0272-4979 (ISSN-L)
  • 1464-3642
OpenAlex
W3004059350
Cited by
126
References
31
Citations by year
2026: 32 · 2025: 30 · 2024: 23 · 2023: 20 · 2022: 12 · 2021: 9
FWCI
5.10
Type
journal article
Language
en
Open access
Locations
  • IMA Journal of Numerical Analysis (publishedVersion, cc-by)
  • Research Explorer (The University of Manchester) (publishedVersion)
  • MIMS EPrints (University of Southampton) (acceptedVersion)
Funders
  • Engineering and Physical Sciences Research Council (EP/P020720/1)
  • MathWorks
  • Royal Society
Topics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Numerical Methods and Algorithms
  • Physical Sciences › Mathematics › Modeling and Simulation › Mathematical and Computational Methods
  • Physical Sciences › Mathematics › Applied Mathematics › Mathematical functions and polynomials
Keywords
softmax, floating-point arithmetic, rounding error analysis, condition number, overflow and underflow, numerical accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review