CoChat

An analog-AI chip for energy-efficient speech recognition and transcription

Open access

The DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index; 12 of the 36 sources its claims cite were checked for support.

Abstract

Abstract Models of artificial intelligence (AI) that have billions of parameters can achieve high accuracy across a range of tasks 1,2 , but they exacerbate the poor energy efficiency of conventional general-purpose processors, such as graphics processing units or central processing units. Analog in-memory computing (analog-AI) 3–7 can provide better energy efficiency by performing matrix–vector multiplications in parallel on ‘memory tiles’. However, analog-AI has yet to demonstrate software-equivalent (SW eq ) accuracy on models that require many such tiles and efficient communication of neural-network activations between the tiles. Here we present an analog-AI chip that combines 35 million phase-change memory devices across 34 tiles, massively parallel inter-tile communication and analog, low-power peripheral circuitry that can achieve up to 12.4 tera-operations per second per watt (TOPS/W) chip-sustained performance. We demonstrate fully end-to-end SW eq accuracy for a small keyword-spotting network and near-SW eq accuracy on the much larger MLPerf 8 recurrent neural-network transducer (RNNT), with more than 45 million weights mapped onto more than 140 million phase-change memory devices across five chips.

Overview

CoChat Summary

## TL;DR

The paper demonstrates a 14-nm analog-AI inference chip for speech recognition and transcription, using 34 tiles of phase-change memory and low-power routing to reach up to 12.4 TOPS/W chip-sustained performance. It reports end-to-end software-equivalent accuracy for keyword spotting and near-software-equivalent accuracy for the much larger MLPerf RNNT model with more than 45 million weights mapped across five chips.

## Key findings

  • The chip combines 35 million phase-change memory devices across 34 tiles and achieves up to 12.4 TOPS/W chip-sustained performance.
  • For keyword spotting, the system achieved 86.14% accuracy, which the paper says is within the MLPerf software-equivalent limit of 85.88% (99% of the original software model’s accuracy); the original software model is reported at 86.75%.
  • For RNNT on Librispeech, the full five-chip system reached 9.475% WER in one configuration, versus a 7.452% software baseline, and with weight expansion improved to 9.258% WER.
  • The RNNT mapping used 45,261,568 of 45,321,309 total network weight and bias parameters in analog memory (99.9%), spread across more than 140 million PCM devices on 5 chips.
  • The paper reports a projected system performance of 6.94 TOPS/W and 546.6 samples/s/W at 3.57 W, and says this is a 14-fold improvement over the best MLPerf result for the task.

## Methods

  • Experimental work used a 14-nm inference chip built from 300-mm wafers, with PCM devices added in back-end-of-line processing; experiments were run with three Xilinx Virtex-7 VC707 FPGAs and x86/MATLAB control.
  • The authors tested two workloads: a keyword-spotting (KWS) model on the Google Speech Commands dataset and the MLPerf RNNT model on Librispeech.
  • Main analyses included hardware-aware training, pruning, calibration, analog weight programming, and weight expansion, plus end-to-end inference measurements of accuracy/WER, power, and TOPS/W; RNNT inference was cascaded across five chips.

## Limitations

  • The paper states that the 5-chip RNNT implementation is not integrated with digital processing; some digital operations are estimated or performed off-chip, so the reported system performance is partly projected rather than fully monolithic.
  • It also states that the chip does not include on-chip digital computing cores or SRAM for auxiliary operations and data staging, which the text makes evident as a limitation for a marketable product.
  • The paper notes that the joint-FC layer and some vector-vector, bias, and nonlinear operations were computed in software/off-chip, so not all of RNNT was done in analog hardware.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-023-06337-5

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Chip architecture. a, Speech recognition has improved markedly over the past 10 years, driving down the WER for both the Librispeech and SwitchBoard (SWB) datasets, thanks to substantial increases in model size and improved networks, such as RNNT or transformer. For comparison with our results, the MLPerf RNNT full-precision WER is shown for two Librispeech datasets (‘test-clean’ and ‘dev-clean’)8, along with this work’s WER, which was computed on Librispeech dev-clean. For model size: B, 1 billion; M, 1 million. b, Inference models are trained using popular frameworks such as PyTorch or TensorFlow. Further optimization for analog AI can be achieved with the IBM analog HW acceleration kit (https://aihwkit.readthedocs.io/en/latest/). c, Trained model weights are then used on a 14-nm chip with 34 analog tiles, two processing elements (PE, not used for this work) and six ILP–OLP pairs. Tiles are labelled as north (N), centre (C) or south (S) followed by west (W) or east (E). d, Each ILP converts 512 8-bit inputs into 512 element vectors of pulse-modulated durations, which are then routed to the analog tiles for integration using a fully parallel 2D mesh that allows multi-casting to multiple tiles. After MAC, the charge on the peripheral capacitors is converted into durations4 and sent either to other tiles, leading to new MACs, or to the OLP, where durations are reconverted into 8-bit representations for off-chip data-processing. e, Transmission Electron Microscopy (TEM) image of one PCM. f, Each tile contains a crossbar array with 512 × 2,048 PCMs, programmed using a parallel row-wise algorithm4. g, PCMs can be organized in a 4-PCM-per-weight configuration, with G+, g+ adding and G−, g− subtracting charge from the peripheral capacitor, with a significance factor F (which is 1 in this paper). h, Alternatively, they can have a 2-PCM-per-weight configuration, which achieves a higher density. By reading different input frames through weights WP1 or WP2, a single tile can map 1,024 × 512 weight layers. i, Finally, two adjacent tiles can share their banks of 512 peripheral capacitors, enabling integration in the analog domain across 2,048 input rows.
Fig. 1 Chip architecture. a, Speech recognition has improved markedly over the past 10 years, driving down the WER for both the Librispeech and SwitchBoard (SWB) datasets, thanks to substantial increases in model size and improved networks, such as RNNT or transformer. For comparison with our results, the MLPerf RNNT full-precision WER is shown for two Librispeech datasets (‘test-clean’ and ‘dev-clean’)8, along with this work’s WER, which was computed on Librispeech dev-clean. For model size: B, 1 billion; M, 1 million. b, Inference models are trained using popular frameworks such as PyTorch or TensorFlow. Further optimization for analog AI can be achieved with the IBM analog HW acceleration kit (https://aihwkit.readthedocs.io/en/latest/). c, Trained model weights are then used on a 14-nm chip with 34 analog tiles, two processing elements (PE, not used for this work) and six ILP–OLP pairs. Tiles are labelled as north (N), centre (C) or south (S) followed by west (W) or east (E). d, Each ILP converts 512 8-bit inputs into 512 element vectors of pulse-modulated durations, which are then routed to the analog tiles for integration using a fully parallel 2D mesh that allows multi-casting to multiple tiles. After MAC, the charge on the peripheral capacitors is converted into durations4 and sent either to other tiles, leading to new MACs, or to the OLP, where durations are reconverted into 8-bit representations for off-chip data-processing. e, Transmission Electron Microscopy (TEM) image of one PCM. f, Each tile contains a crossbar array with 512 × 2,048 PCMs, programmed using a parallel row-wise algorithm4. g, PCMs can be organized in a 4-PCM-per-weight configuration, with G+, g+ adding and G−, g− subtracting charge from the peripheral capacitor, with a significance factor F (which is 1 in this paper). h, Alternatively, they can have a 2-PCM-per-weight configuration, which achieves a higher density. By reading different input frames through weights WP1 or WP2, a single tile can map 1,024 × 512 weight layers. i, Finally, two adjacent tiles can share their banks of 512 peripheral capacitors, enabling integration in the analog domain across 2,048 input rows.CC BY · PubMed Central
Fig. 2 Reconfigurable architecture and routing. a, For maximum test-time flexibility, each tile contains a user-programmable LC that defines all timing sequences. b, LC controls signal routing through the 2D mesh from ILP to the tile, MAC, output duration generation through ramp plus comparator circuitry and borderguard tile routing configuration. c, As an example, LC can implement 2D-mesh concatenation such as merging the durations originating from the even columns on one tile with the durations coming from the odd columns of another tile. d, To test the communication, 1 million random input durations are multi-cast, in parallel, to all 6 OLPs. Durations randomly vary between 0 and 50 ns (dark-blue lines) or between 0 and either 100, 150, 200 or 250 ns (lighter shades of blue) with 1-ns granularity. Cumulative distribution functions (CDFs) reveal that the communication error never exceeds 5 ns, demonstrating high transport accuracy.
Fig. 2 Reconfigurable architecture and routing. a, For maximum test-time flexibility, each tile contains a user-programmable LC that defines all timing sequences. b, LC controls signal routing through the 2D mesh from ILP to the tile, MAC, output duration generation through ramp plus comparator circuitry and borderguard tile routing configuration. c, As an example, LC can implement 2D-mesh concatenation such as merging the durations originating from the even columns on one tile with the durations coming from the odd columns of another tile. d, To test the communication, 1 million random input durations are multi-cast, in parallel, to all 6 OLPs. Durations randomly vary between 0 and 50 ns (dark-blue lines) or between 0 and either 100, 150, 200 or 250 ns (lighter shades of blue) with 1-ns granularity. Cumulative distribution functions (CDFs) reveal that the communication error never exceeds 5 ns, demonstrating high transport accuracy.CC BY · PubMed Central
Fig. 3 End-to-end KWS task. a, To classify spoken words into one of the 12 highlighted classes for KWS, an FC baseline is used as a reference. b, The network is then retrained using HWA techniques. c, The end-to-end implementation uses four analog tiles. d, An AB method is used to increase MAC accuracy. Weights W and −W are programmed on WP1 and WP2 respectively. By inferencing the desired input x on WP1 and then −x on WP2, the MAC is collected twice (xw + (−x) × (−w)), cancelling out any fixed peripheral circuitry asymmetries and improving MAC accuracy. e, A timing diagram shows that a full frame is processed in 2.4 μs. Because the ReLU activation (implemented on-chip in the analog domain) generates positive-only outputs, the second layer requires only two integration steps, rather than the four needed in the first layer. f, Experimental activations after layers L0, L1 and output correlate closely with ideal SW MACs calculated using HW input. PDF, probability distribution function. g, This leads to SWeq accuracy for this fully end-to-end demonstration.
Fig. 3 End-to-end KWS task. a, To classify spoken words into one of the 12 highlighted classes for KWS, an FC baseline is used as a reference. b, The network is then retrained using HWA techniques. c, The end-to-end implementation uses four analog tiles. d, An AB method is used to increase MAC accuracy. Weights W and −W are programmed on WP1 and WP2 respectively. By inferencing the desired input x on WP1 and then −x on WP2, the MAC is collected twice (xw + (−x) × (−w)), cancelling out any fixed peripheral circuitry asymmetries and improving MAC accuracy. e, A timing diagram shows that a full frame is processed in 2.4 μs. Because the ReLU activation (implemented on-chip in the analog domain) generates positive-only outputs, the second layer requires only two integration steps, rather than the four needed in the first layer. f, Experimental activations after layers L0, L1 and output correlate closely with ideal SW MACs calculated using HW input. PDF, probability distribution function. g, This leads to SWeq accuracy for this fully end-to-end demonstration.CC BY · PubMed Central
Fig. 4 MLPerf RNNT network for speech transcription. a, MLPerf RNNT model, trained on the Librispeech dataset, comprises encoder (Enc), decoder (Dec) and joint blocks. The input signal is digitally preprocessed and stacked to generate the input of Enc-LSTM0 (chip 1) and Enc-LSTM1 (chip 2). The resulting output vectors are again time-stacked before feeding a 2,048-input Enc-LSTM2 (chip 3), followed by two 1,024-input Enc-LSTM3,4 and an Enc-FC linear layer (chip 4). The resulting encoder output is then merged with the vectors received from the Dec (chip 5). Finally, a joint-FC calculates the next-letter probability (in SW), which feeds back to the Dec. This entails greedy decoding in which the highest probability selects the output letter. b, SW-based sensitivity analysis performed by progressively quantizing the FP32 MLPerf weights. c, The WER increases beyond the SWeq limit when weights are excessively quantized. d, There is a threshold nbits at which the WER is still SWeq for the full network, the full network without joint-FC quantization, and for each individual layer. While Dec-LSTM1 is the most resilient to noise, joint-FC exhibits significant sensitivity and is small in size, so it is not mapped in analog to preserve high accuracy. e, All the other layers are mapped to analog tiles (mapping details in Extended Data Figs. 5, 7). All arrows show the input signal routing and are operating at the same time, each performing a simultaneous multi-cast to all tiles that show the same-colour MAC arrow. Note that the borderguard circuits can enable duration data arriving at the west side of a tile to deliver durations onto the rows of that tile, and a completely different duration-vector passes over the centre of that tile on its routing wires at the same time. Small arrows indicate how MACs are aggregated in the analog domain across tile pairs. f, The output duration routing. Each arrow colour requires its own time slot: three for chips 1, 2, 3 and 4, and one for chip 5. Output routing from tiles to OLPs can involve implicit concatenation (chip 5). More details are given in the Methods. The joint block and all LSTM vector–vector operations are computed off-chip. g, More than 45 million weights are mapped using more than 140 million PCMs, with an average of 2.9 (3.1 with Wexp) PCMs per weight. Coloured bars show PCMs, white bars show weights.
Fig. 4 MLPerf RNNT network for speech transcription. a, MLPerf RNNT model, trained on the Librispeech dataset, comprises encoder (Enc), decoder (Dec) and joint blocks. The input signal is digitally preprocessed and stacked to generate the input of Enc-LSTM0 (chip 1) and Enc-LSTM1 (chip 2). The resulting output vectors are again time-stacked before feeding a 2,048-input Enc-LSTM2 (chip 3), followed by two 1,024-input Enc-LSTM3,4 and an Enc-FC linear layer (chip 4). The resulting encoder output is then merged with the vectors received from the Dec (chip 5). Finally, a joint-FC calculates the next-letter probability (in SW), which feeds back to the Dec. This entails greedy decoding in which the highest probability selects the output letter. b, SW-based sensitivity analysis performed by progressively quantizing the FP32 MLPerf weights. c, The WER increases beyond the SWeq limit when weights are excessively quantized. d, There is a threshold nbits at which the WER is still SWeq for the full network, the full network without joint-FC quantization, and for each individual layer. While Dec-LSTM1 is the most resilient to noise, joint-FC exhibits significant sensitivity and is small in size, so it is not mapped in analog to preserve high accuracy. e, All the other layers are mapped to analog tiles (mapping details in Extended Data Figs. 5, 7). All arrows show the input signal routing and are operating at the same time, each performing a simultaneous multi-cast to all tiles that show the same-colour MAC arrow. Note that the borderguard circuits can enable duration data arriving at the west side of a tile to deliver durations onto the rows of that tile, and a completely different duration-vector passes over the centre of that tile on its routing wires at the same time. Small arrows indicate how MACs are aggregated in the analog domain across tile pairs. f, The output duration routing. Each arrow colour requires its own time slot: three for chips 1, 2, 3 and 4, and one for chip 5. Output routing from tiles to OLPs can involve implicit concatenation (chip 5). More details are given in the Methods. The joint block and all LSTM vector–vector operations are computed off-chip. g, More than 45 million weights are mapped using more than 140 million PCMs, with an average of 2.9 (3.1 with Wexp) PCMs per weight. Coloured bars show PCMs, white bars show weights.CC BY · PubMed Central
Fig. 5 Experimental WER using Librispeech on MLPerf RNNT. a, Single-layer WER. The graph shows an experimental sensitivity analysis obtained by implementing one layer on-chip and all the others in SW at FP32 precision. The most critical layer is Enc-LSTM0. b, Cumulative WER. Full RNNT inference using all five chips on the full Librispeech validation dataset. The bars from left to right show the cumulative WER obtained when implementing increasingly more layers on-chip. The full RNNT WER, using the original MLPerf weights, achieved across five chips (right-most bar) is 9.475%. c, After one week of PCM drift, the cumulative WER slightly increases to 9.894%, just 0.4% more than day-0 WER. d, To further improve the accuracy, a weight-expansion technique is introduced for Enc-LSTM0. Given a MAC Wx × x, the insertion of a random normal matrix M and its pseudoinverse pinv(M) leads to the same MAC output. However, now Wx2 = Wx × pinv(M) contains more rows N, with an increased signal-to-noise ratio. Whereas signal increases linearly with N, the aggregate noise across the larger number of rows increases sub-linearly (∝√N if noise sources are independent Gaussians). e, Simulation results. When quantizing Enc-LSTM0 to nbits = 3.5 bits, the WER is 42%. Weight expansion greatly improves the resilience, even for only slightly expanded Wx2 matrices, with the WER reduced to 7.9%, well below SWeq. f, Similar accuracy benefits are observed experimentally when implementing weight expansion on Enc-LSTM0 on-chip, revealing stronger WER reduction with respect to weight averaging. M × x is digitally preprocessed. Wx2 expansion to 1,024 rows enables a 9.258% WER on the full RNNT, 1.81% from the SW baseline, 0.88% from SWeq.
Fig. 5 Experimental WER using Librispeech on MLPerf RNNT. a, Single-layer WER. The graph shows an experimental sensitivity analysis obtained by implementing one layer on-chip and all the others in SW at FP32 precision. The most critical layer is Enc-LSTM0. b, Cumulative WER. Full RNNT inference using all five chips on the full Librispeech validation dataset. The bars from left to right show the cumulative WER obtained when implementing increasingly more layers on-chip. The full RNNT WER, using the original MLPerf weights, achieved across five chips (right-most bar) is 9.475%. c, After one week of PCM drift, the cumulative WER slightly increases to 9.894%, just 0.4% more than day-0 WER. d, To further improve the accuracy, a weight-expansion technique is introduced for Enc-LSTM0. Given a MAC Wx × x, the insertion of a random normal matrix M and its pseudoinverse pinv(M) leads to the same MAC output. However, now Wx2 = Wx × pinv(M) contains more rows N, with an increased signal-to-noise ratio. Whereas signal increases linearly with N, the aggregate noise across the larger number of rows increases sub-linearly (∝√N if noise sources are independent Gaussians). e, Simulation results. When quantizing Enc-LSTM0 to nbits = 3.5 bits, the WER is 42%. Weight expansion greatly improves the resilience, even for only slightly expanded Wx2 matrices, with the WER reduced to 7.9%, well below SWeq. f, Similar accuracy benefits are observed experimentally when implementing weight expansion on Enc-LSTM0 on-chip, revealing stronger WER reduction with respect to weight averaging. M × x is digitally preprocessed. Wx2 expansion to 1,024 rows enables a 9.258% WER on the full RNNT, 1.81% from the SW baseline, 0.88% from SWeq.CC BY · PubMed Central
Fig. 6 MLPerf RNNT power and system performance. a, Measured power and TOPS/W are shown for each chip. TOPS/W (coloured bars) correlate with the number of weights used on each chip (white bars). D, duration; DAC, digital-to-analog converter; PLL, phase-locked loop. b, Reducing the maximum input duration leads to an improvement in TOPS/W with only a small amount of WER degradation (chip 4 is measured, other layers in SW at FP32). c, Energy efficiency at various levels: analog integration only (1.5 V power domain), full chip, all 5 chips for RNNT (analog integration only and full chip), and full system level including estimated digital processing energy20. d, Simulated performance for an integrated system shows that the average processing time for each sample is 500 μs, more than 104 times faster than the input speech sentence, thus enabling real-time transcription. Total processing time = 1.29 s and total real audio = 4 h 20 min, so the real-time factor ≈ 8 × 10−5 ≪ 1. e, Number of operations performed on-chip versus off-chip in the RNNT experiment, with a 325:1 ratio for the original MLPerf weights (Wx) and 88:1 with weight expansion (Wx2) (Fig. 5d). f, Samples per second per watt and TOPS/W performance for comparison with MLPerf submissions, showing a 14-fold improvement for our system.
Fig. 6 MLPerf RNNT power and system performance. a, Measured power and TOPS/W are shown for each chip. TOPS/W (coloured bars) correlate with the number of weights used on each chip (white bars). D, duration; DAC, digital-to-analog converter; PLL, phase-locked loop. b, Reducing the maximum input duration leads to an improvement in TOPS/W with only a small amount of WER degradation (chip 4 is measured, other layers in SW at FP32). c, Energy efficiency at various levels: analog integration only (1.5 V power domain), full chip, all 5 chips for RNNT (analog integration only and full chip), and full system level including estimated digital processing energy20. d, Simulated performance for an integrated system shows that the average processing time for each sample is 500 μs, more than 104 times faster than the input speech sentence, thus enabling real-time transcription. Total processing time = 1.29 s and total real audio = 4 h 20 min, so the real-time factor ≈ 8 × 10−5 ≪ 1. e, Number of operations performed on-chip versus off-chip in the RNNT experiment, with a 325:1 ratio for the original MLPerf weights (Wx) and 88:1 with weight expansion (Wx2) (Fig. 5d). f, Samples per second per watt and TOPS/W performance for comparison with MLPerf submissions, showing a 14-fold improvement for our system.CC BY · PubMed Central
Extended Data Fig. 1 Analog in-memory computing chip and test platform. (a) Packaged modules used in the experiments. (b) Micrograph of the chip at H3 metal level, showing all analog tiles and the corresponding area breakdown. (c) The test platform is based on a chip socket mounted on a custom board driven by 3 Xilinx Virtex-7 VC707 FPGAs. Power is supplied by four Keysight E36312A. The three FPGAs are connected through the board, and the overall system is controlled by an x86 machine using a PCIe connector. (d) For routing precision, end-to-end KWS, and RNNT power measurements, the chip is run without any need to send intermediate data back to an x86 machine during the experiment. KWS models are trained for chip deployment using HWA methods. For RNNT accuracy measurements, intermediate data moves between the chip and an x86 machine. To predict the RNNT performance, we assume digital circuitry would be integrated on the chip next to the ILP/OLP pair, leading to predictions that should be very close to a fully integrated chip in which both analog tiles and digital compute cores are tightly integrated20.
Extended Data Fig. 1 Analog in-memory computing chip and test platform. (a) Packaged modules used in the experiments. (b) Micrograph of the chip at H3 metal level, showing all analog tiles and the corresponding area breakdown. (c) The test platform is based on a chip socket mounted on a custom board driven by 3 Xilinx Virtex-7 VC707 FPGAs. Power is supplied by four Keysight E36312A. The three FPGAs are connected through the board, and the overall system is controlled by an x86 machine using a PCIe connector. (d) For routing precision, end-to-end KWS, and RNNT power measurements, the chip is run without any need to send intermediate data back to an x86 machine during the experiment. KWS models are trained for chip deployment using HWA methods. For RNNT accuracy measurements, intermediate data moves between the chip and an x86 machine. To predict the RNNT performance, we assume digital circuitry would be integrated on the chip next to the ILP/OLP pair, leading to predictions that should be very close to a fully integrated chip in which both analog tiles and digital compute cores are tightly integrated20.CC BY · PubMed Central
Extended Data Fig. 2 Analog circuits and routing configurations. (a) Input Landing Pad circuitry receives an 8-bit word, which is converted to a duration using a counter. Output Landing Pad circuitry receives one duration and converts it into an 8-bit word using 512 counters plus falling edge detectors. (b) Each tile has 512x512 unit cells, each consists of 4 PCM devices controlled by Word Lines and Select Lines driven by the tile West circuitry. During weight programming, Return Lines (RL) are driven at higher voltage (max 3V) and VSIG are grounded to enable PCM write. (c) During inference, in the integration phase, RL are grounded and one or both VSIG are biased at read voltage Vread = 0.3 V using a per-column operational amplifier. Current is then mirrored into a per-column capacitor. (d) For duration generation phase, capacitor voltage is compared against a shared and tunable ramp-voltage. The output of the comparator is a pulse with duration proportional to the voltage on the capacitor, gated by an EN signal using an AND gate. By properly tuning the timing of this EN signal with the Local Controller, activation functions such as ReLU or hard sigmoid can be implemented. (e) 512 durations are then sent into the parallel 2D mesh using a per-column South-North routing circuit. (This same circuit is also used to send pulses onto each column for PCM programming purposes). (f) Finally, West-East Routing enables durations to access another tile and/or to simply move across the tile. Since each borderguard can independently block or pass signals, complex routing patterns including multi-cast and even concatenation are supported.
Extended Data Fig. 2 Analog circuits and routing configurations. (a) Input Landing Pad circuitry receives an 8-bit word, which is converted to a duration using a counter. Output Landing Pad circuitry receives one duration and converts it into an 8-bit word using 512 counters plus falling edge detectors. (b) Each tile has 512x512 unit cells, each consists of 4 PCM devices controlled by Word Lines and Select Lines driven by the tile West circuitry. During weight programming, Return Lines (RL) are driven at higher voltage (max 3V) and VSIG are grounded to enable PCM write. (c) During inference, in the integration phase, RL are grounded and one or both VSIG are biased at read voltage Vread = 0.3 V using a per-column operational amplifier. Current is then mirrored into a per-column capacitor. (d) For duration generation phase, capacitor voltage is compared against a shared and tunable ramp-voltage. The output of the comparator is a pulse with duration proportional to the voltage on the capacitor, gated by an EN signal using an AND gate. By properly tuning the timing of this EN signal with the Local Controller, activation functions such as ReLU or hard sigmoid can be implemented. (e) 512 durations are then sent into the parallel 2D mesh using a per-column South-North routing circuit. (This same circuit is also used to send pulses onto each column for PCM programming purposes). (f) Finally, West-East Routing enables durations to access another tile and/or to simply move across the tile. Since each borderguard can independently block or pass signals, complex routing patterns including multi-cast and even concatenation are supported.CC BY · PubMed Central
Extended Data Fig. 3 Keyword Spotting (KWS) Pre-processing and calibration. Hardware-aware (HWA) training was applied to improve model robustness against hardware imperfections, primarily due to programming errors. (a) Weight programming correlation for the HWA model. (b) Dependence of model accuracy on injected noise on weights and intermediate activations during training. (c) Mel-frequency cepstral coefficients (MFCC), representing the fingerprint for each keyword, are flattened and truncated before input to the fully-connected network. (d) The full input would require 1960 rows, however, to reduce the model to 1024-inputs, pruning is performed by removing all the inputs exhibiting a mean absolute validation input value lower than the indicated threshold. (e) The table shows a comparison of the KWS models and accuracies. (f) Since KWS is fully end-to-end on-chip, an on-chip calibration process is performed at the tile, leveraging 8 additional PCM bias rows to shift the MAC up/down to compensate for any intrinsic column-wise offsets. Slope of the MACs is compensated by re-scaling weights per-column. Calibration is performed using validation input data; inference results are reported for the test dataset.
Extended Data Fig. 3 Keyword Spotting (KWS) Pre-processing and calibration. Hardware-aware (HWA) training was applied to improve model robustness against hardware imperfections, primarily due to programming errors. (a) Weight programming correlation for the HWA model. (b) Dependence of model accuracy on injected noise on weights and intermediate activations during training. (c) Mel-frequency cepstral coefficients (MFCC), representing the fingerprint for each keyword, are flattened and truncated before input to the fully-connected network. (d) The full input would require 1960 rows, however, to reduce the model to 1024-inputs, pruning is performed by removing all the inputs exhibiting a mean absolute validation input value lower than the indicated threshold. (e) The table shows a comparison of the KWS models and accuracies. (f) Since KWS is fully end-to-end on-chip, an on-chip calibration process is performed at the tile, leveraging 8 additional PCM bias rows to shift the MAC up/down to compensate for any intrinsic column-wise offsets. Slope of the MACs is compensated by re-scaling weights per-column. Calibration is performed using validation input data; inference results are reported for the test dataset.CC BY · PubMed Central
Extended Data Fig. 4 Experimental MLPerf RNNT weight programming. Experimental correlation between target and programmed weights on chip-1 over 32 tiles for both (a) WP1 and (b) WP2. (c),(d) The corresponding probability distribution functions (PDF) of errors, expressed as percentage of the maximum weight, reveal high-yield chips with very few erroneous weights. (e) Table showing the analog yield, or the fraction of weights with programming error within 20% of the maximum weight. After integration of the PDFs in (c,d), the corresponding cumulative distribution functions (CDFs) are computed and the 1%-99% spread is collected, providing 2 data points (one for WP1 and one for WP2) for each tile. The plot shows the corresponding CDFs for each of the five chips used in RNNT experiments. To control the peripheral circuitry saturation, some tiles have weights mapped into a smaller conductance range (Max W equal to 80), leading to a different 1%-99% spread, e.g. the points with increased spread on chip-1 CDF in (e).
Extended Data Fig. 4 Experimental MLPerf RNNT weight programming. Experimental correlation between target and programmed weights on chip-1 over 32 tiles for both (a) WP1 and (b) WP2. (c),(d) The corresponding probability distribution functions (PDF) of errors, expressed as percentage of the maximum weight, reveal high-yield chips with very few erroneous weights. (e) Table showing the analog yield, or the fraction of weights with programming error within 20% of the maximum weight. After integration of the PDFs in (c,d), the corresponding cumulative distribution functions (CDFs) are computed and the 1%-99% spread is collected, providing 2 data points (one for WP1 and one for WP2) for each tile. The plot shows the corresponding CDFs for each of the five chips used in RNNT experiments. To control the peripheral circuitry saturation, some tiles have weights mapped into a smaller conductance range (Max W equal to 80), leading to a different 1%-99% spread, e.g. the points with increased spread on chip-1 CDF in (e).CC BY · PubMed Central
Extended Data Fig. 5 MLPerf RNNT Encoder mapping and signal routing on chip. (a) RNNT Encoder LSTMs weights are represented by two large matrices, Wx which multiplies the LSTM input, x, and Wh which multiplies the LSTM recurrent signal, h. Depending on the sizes of x and h, a variable number of tiles is required. Both conventional and weight expansion mappings are shown for Enc-LSTM0. In addition, Enc-LSTM0, Enc-LSTM1 and the Wh of Enc-LSTM2 implement Asymmetry Balance. Tiles connected with a dark blue line have shared capacitors, enabling 2048-wide analog MAC. In Enc-LSTM0, Enc-LSTM1 and Enc-LSTM2, MACs from tiles 1,9 and 2,10 are summed in digital, while all the other pairs are summed on-chip in analog. (b) Every analog MAC on the encoder chips requires seven 300 ns time steps to process, including digitization of the output. During the first four steps, MAC operations are performed, providing input signals from ILPs as indicated in the figure. In cases where AB is used (Enc LSTM0, Enc LSTM1, Wh portion of Enc LSTM2), opposite-signed inputs are provided in two of the four MAC time steps. During the last three time steps, MAC results are sent out to the OLPs.
Extended Data Fig. 5 MLPerf RNNT Encoder mapping and signal routing on chip. (a) RNNT Encoder LSTMs weights are represented by two large matrices, Wx which multiplies the LSTM input, x, and Wh which multiplies the LSTM recurrent signal, h. Depending on the sizes of x and h, a variable number of tiles is required. Both conventional and weight expansion mappings are shown for Enc-LSTM0. In addition, Enc-LSTM0, Enc-LSTM1 and the Wh of Enc-LSTM2 implement Asymmetry Balance. Tiles connected with a dark blue line have shared capacitors, enabling 2048-wide analog MAC. In Enc-LSTM0, Enc-LSTM1 and Enc-LSTM2, MACs from tiles 1,9 and 2,10 are summed in digital, while all the other pairs are summed on-chip in analog. (b) Every analog MAC on the encoder chips requires seven 300 ns time steps to process, including digitization of the output. During the first four steps, MAC operations are performed, providing input signals from ILPs as indicated in the figure. In cases where AB is used (Enc LSTM0, Enc LSTM1, Wh portion of Enc LSTM2), opposite-signed inputs are provided in two of the four MAC time steps. During the last three time steps, MAC results are sent out to the OLPs.CC BY · PubMed Central
Extended Data Fig. 6 MLPerf RNNT Enc-LSTM0 input processing. Due to the large sensitivity of the first Encoder LSTM Wx matrix, input data is digitally pre-processed to increase the MAC signal-to-noise ratio. (a) Using the original MLPerf weights, input data is first shifted to zero-mean and normalized to a common maximum input amplitude. Then, on-chip MAC is performed, followed by MAC digital denormalization and addition of mean Input*ΣW. This last operation collapses to a simple single number (mean Input) times one 4096-vector represented by the column-wise sum of weights, not requiring a full digital 1024*4096 MAC operation. Only after these steps the affine coefficients (slope and offset correction) are applied. (b) When using expanded weights, the mean removal is no longer needed since the random matrix M introduced in the main paper has zero mean, leading to naturally near zero-mean input values, thus simplifying input pre-processing. (c) All the other RNNT blocks (rest of Encoder and Decoder) only apply affine coefficients, without any normalization or mean removal. (d) Data-type conversion during the MLPerf RNNT inference: software computation is performed in FP32. Before access to the chip, data is converted into a pair of UINT8 vectors, one for each polarity of the incoming activations. This data is loaded on-chip in the ILP, then converted into time duration and used as MAC input (‘negative’ inputs are durations sent during the second and fourth integration steps, as shown in Extended Data Figs. 5b and 7d). MAC output is represented as analog voltage on a capacitor, then converted into time duration by the peripheral circuitry (details in Extended Data Fig. 2d). Finally, the OLP converts durations into UINT8. Data is sent off-chip and converted in FP32 during the calibration phase. (e) Overall view of the equations solved in the RNNT: essentially, all MACs are performed on chip, while all vector-vector operations, non-linear activations and biases are computed in software. The joint layer is implemented in software, as explained in Fig. 4.
Extended Data Fig. 6 MLPerf RNNT Enc-LSTM0 input processing. Due to the large sensitivity of the first Encoder LSTM Wx matrix, input data is digitally pre-processed to increase the MAC signal-to-noise ratio. (a) Using the original MLPerf weights, input data is first shifted to zero-mean and normalized to a common maximum input amplitude. Then, on-chip MAC is performed, followed by MAC digital denormalization and addition of mean Input*ΣW. This last operation collapses to a simple single number (mean Input) times one 4096-vector represented by the column-wise sum of weights, not requiring a full digital 1024*4096 MAC operation. Only after these steps the affine coefficients (slope and offset correction) are applied. (b) When using expanded weights, the mean removal is no longer needed since the random matrix M introduced in the main paper has zero mean, leading to naturally near zero-mean input values, thus simplifying input pre-processing. (c) All the other RNNT blocks (rest of Encoder and Decoder) only apply affine coefficients, without any normalization or mean removal. (d) Data-type conversion during the MLPerf RNNT inference: software computation is performed in FP32. Before access to the chip, data is converted into a pair of UINT8 vectors, one for each polarity of the incoming activations. This data is loaded on-chip in the ILP, then converted into time duration and used as MAC input (‘negative’ inputs are durations sent during the second and fourth integration steps, as shown in Extended Data Figs. 5b and 7d). MAC output is represented as analog voltage on a capacitor, then converted into time duration by the peripheral circuitry (details in Extended Data Fig. 2d). Finally, the OLP converts durations into UINT8. Data is sent off-chip and converted in FP32 during the calibration phase. (e) Overall view of the equations solved in the RNNT: essentially, all MACs are performed on chip, while all vector-vector operations, non-linear activations and biases are computed in software. The joint layer is implemented in software, as explained in Fig. 4.CC BY · PubMed Central
Extended Data Fig. 7 MLPerf RNNT Decoder mapping and signal routing on chip. Instead of processing Embedding Emb and Dec-LSTM0 Wx layers separately, we first compress in one single matrix the product Emb * Wx. At this point, the first Dec-LSTM0 shows Emb * Wx matrix with a (28*1280) size, contrasting with the (320*1280) Wh size to sum directly in analog. To balance signal magnitude, nine copies of Emb * (Wx/9) are programmed, achieving comparable weight absolute magnitudes. (b) Weight mapping and (c) signal routing implementing Asymmetry Balance. Routing from tiles to Output Landing Pads utilizes implicit vector concatenation on the 2D mesh, enabling more efficient data transport. (d) The processing of one full frame requires 5 time steps of 300 ns each.
Extended Data Fig. 7 MLPerf RNNT Decoder mapping and signal routing on chip. Instead of processing Embedding Emb and Dec-LSTM0 Wx layers separately, we first compress in one single matrix the product Emb * Wx. At this point, the first Dec-LSTM0 shows Emb * Wx matrix with a (28*1280) size, contrasting with the (320*1280) Wh size to sum directly in analog. To balance signal magnitude, nine copies of Emb * (Wx/9) are programmed, achieving comparable weight absolute magnitudes. (b) Weight mapping and (c) signal routing implementing Asymmetry Balance. Routing from tiles to Output Landing Pads utilizes implicit vector concatenation on the 2D mesh, enabling more efficient data transport. (d) The processing of one full frame requires 5 time steps of 300 ns each.CC BY · PubMed Central
Extended Data Fig. 8 Experimental MLPerf RNNT MAC. (a) Error histogram and (b) correlation between experimental and target MAC are shown for every chip used during the RNNT inference experiment. On each correlation, white dotted lines highlight the main region of interest, since MACs are followed by sigmoid, tanh or ReLU which naturally filter out portions of MAC. The spread, σ, is calculated only within the highlighted Regions Of Interest (ROI). Data from both original Enc-LSTM0 and weight-expanded Enc-LSTM0 are reported, showing a better sigma for the weight expanded case. Enc-LSTM2, Enc-LSTM3, and Enc-LSTM4 show larger spread due to partial (Enc-LSTM2) or no (Enc-LSTM3, Enc-LSTM4) application of Asymmetry Balance. In addition, Enc-LSTM2 MAC is calculated on larger (3072 instead of 2048) inputs. Finally, decoder layers show larger σ, maybe caused by higher capacitor/Output Landing Pad saturation effects, which however have little impact on the overall WER, as revealed by the accuracy results in the main paper (Fig. 5a,b).
Extended Data Fig. 8 Experimental MLPerf RNNT MAC. (a) Error histogram and (b) correlation between experimental and target MAC are shown for every chip used during the RNNT inference experiment. On each correlation, white dotted lines highlight the main region of interest, since MACs are followed by sigmoid, tanh or ReLU which naturally filter out portions of MAC. The spread, σ, is calculated only within the highlighted Regions Of Interest (ROI). Data from both original Enc-LSTM0 and weight-expanded Enc-LSTM0 are reported, showing a better sigma for the weight expanded case. Enc-LSTM2, Enc-LSTM3, and Enc-LSTM4 show larger spread due to partial (Enc-LSTM2) or no (Enc-LSTM3, Enc-LSTM4) application of Asymmetry Balance. In addition, Enc-LSTM2 MAC is calculated on larger (3072 instead of 2048) inputs. Finally, decoder layers show larger σ, maybe caused by higher capacitor/Output Landing Pad saturation effects, which however have little impact on the overall WER, as revealed by the accuracy results in the main paper (Fig. 5a,b).CC BY · PubMed Central
Extended Data Fig. 9 Librispeech transcribed sentences. First ten transcribed sentences from the Librispeech validation dataset. The first line shows the reference sentence, the second line corresponds to the MLPerf baseline prediction (WER 7.452%), and the third line shows the sentences produced by our five-chips experiment including the weight expansion technique (WER 9.258%).
Extended Data Fig. 9 Librispeech transcribed sentences. First ten transcribed sentences from the Librispeech validation dataset. The first line shows the reference sentence, the second line corresponds to the MLPerf baseline prediction (WER 7.452%), and the third line shows the sentences produced by our five-chips experiment including the weight expansion technique (WER 9.258%).CC BY · PubMed Central
Extended Data Fig. 10 System performance estimation. To calculate processing time for RNNT on an integrated system as described in Fig. 6, (a) a simulator based on the MAC runtime on the actual chip and plausible digital processing is considered based on (b) specific timing assumptions stemming from our experiment and prior architectural work20. (c) Detailed breakdown of operations and energy across the 5 chips, including additional digital operations required to process activations from chips. (d) Total on-chip and off-chip number of operations and energy, including measured analog operations (this paper) or estimates for digital ops20. (e) Comparison with MLPerf submissions on RNNT shows a 14 × advantage in energy-efficiency.
Extended Data Fig. 10 System performance estimation. To calculate processing time for RNNT on an integrated system as described in Fig. 6, (a) a simulator based on the MAC runtime on the actual chip and plausible digital processing is considered based on (b) specific timing assumptions stemming from our experiment and prior architectural work20. (c) Detailed breakdown of operations and energy across the 5 chips, including additional digital operations required to process activations from chips. (d) Total on-chip and off-chip number of operations and energy, including measured analog operations (this paper) or estimates for digital ops20. (e) Comparison with MLPerf submissions on RNNT shows a 14 × advantage in energy-efficiency.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 1 Not valid · 8 Not checked · 103 Own results and interpretations

48 more claims

Create a free account to see all 116 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 43
  • Probably not a substantial claim. 24
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 45 references

Cited by

See all 269

Metadata

Authors
Ambrogio, S., Narayanan, P., Okazaki, A., Fasoli, A., Mackin, C., Hosokawa, K., Nomura, A., Yasuda, T., Chen, A., Friz, A., Ishii, M., Luquin, J., Kohda, Y., Saulnier, N., Brew, K., Choi, S., Ok, I., Philip, T., Chan, V., Silvestre, C., Ahsan, I., Narayanan, V., Tsai, H., & Burr, G. W.
Affiliations
  • IBM Research - Almaden (US)
  • IBM Research - Tokyo (JP)
  • IBM Research - Albany (US)
  • IBM Research - Thomas J. Watson Research Center (US)
Corresponding author
Ambrogio, S.
Venue
Nature
Published
2023-08-23
Volume
620
Issue
7975
Pages
768-775
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W4386100686
Cited by
269
References
46
Citations by year
2026: 85 · 2025: 105 · 2024: 71 · 2023: 8
FWCI
19.32
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Analog and Mixed-Signal Circuit Design
Keywords
in-memory computing, phase-change memory, energy-efficient computing, speech recognition, speech transcription, RNN-T, keyword spotting, matrix-vector multiplication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An analog-AI chip for energy-efficient speech recognition and transcription. https://app.cochat.ai/papers/10.1038/s41586-023-06337-5
The paper
Ambrogio, S., Narayanan, P., Okazaki, A., Fasoli, A., Mackin, C., Hosokawa, K., Nomura, A., Yasuda, T., Chen, A., Friz, A., Ishii, M., Luquin, J., Kohda, Y., Saulnier, N., Brew, K., Choi, S., Ok, I., Philip, T., Chan, V., . . . Burr, G. W. (2023). An analog-AI chip for energy-efficient speech recognition and transcription. Nature. https://doi.org/10.1038/s41586-023-06337-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.