CoChat

QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

Open access
Li, Ang, Stein, Samuel, Krishnamoorthy, Sriram, Ang, James · 2023 · ACM Transactions on Quantum Computing

The DOI matches its Crossref record; 134 of the 134 references it cites were found in a registry or an index; 11 of the 51 sources its claims cite were checked for support.

Abstract

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and so on, trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, comprising 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench .

Overview

CoChat Summary

## TL;DR

QASMBench is an OpenQASM-based benchmark suite for evaluating and simulating NISQ quantum devices, compilers, and simulators using real quantum workloads from many domains. The paper also introduces four circuit metrics and reports fidelity experiments on 12 IBM-Q machines plus cross-vendor comparisons on IonQ and Rigetti systems.

## Key findings

  • QASMBench contains benchmark circuits organized into small (2–5 qubits), medium (6–15 qubits), and large (>15 qubits) categories; the largest circuit is `adder_n127` with 3991 gates and 127 qubits.
  • The authors propose four circuit metrics: gate density, retention lifespan, measurement density, and entanglement variance, in addition to width and depth, to characterize execution efficiency and NISQ sensitivity.
  • In their IBM-Q evaluation, they ran 24 circuits from the small category on 12 IBM-Q machines, using density matrix state tomography, for 25,000 circuit evaluations total and 8192 shots per evaluation.
  • They report that fidelity is generally higher for shallower circuits, and that larger machines do not necessarily give higher fidelity; for example, the paper says shallow circuits generally have higher fidelity than deep ones, and the 127-qubit IBM-Q Washington showed reduced fidelity compared with smaller machines.
  • In cross-vendor tests on 4 circuits (Deutsch, DNN, Grover, iSWAP), IonQ’s trapped-ion QPU generally had higher fidelity for relatively shallow circuits, but performance dropped for deeper circuits; the paper attributes this to IonQ’s 99.35% single-gate fidelity versus 99.90% for IBM-Q Washington.

## Methods

  • The paper presents a benchmark suite built around OpenQASM circuits, drawing routines from multiple domains including chemistry, simulation, linear algebra, search, optimization, arithmetic, machine learning, fault tolerance, cryptography, and quantum communication.
  • It categorizes benchmarks by qubit count into small, medium, and large groups and reports circuit properties such as qubits, gate counts, and CX counts for each benchmark.
  • For evaluation, it uses fidelity defined via the distance between the ideal state from a noiseless classical simulation and the reconstructed noisy state from density matrix tomography; it also compares results across IBM-Q, IonQ, and Rigetti devices.

## Limitations

  • The paper states that density matrix tomography is extremely computationally expensive and requires 3^n circuit evaluations for an n-qubit circuit, making it feasible only for small circuits.
  • It also states that tomography must be performed within a short calibration window; otherwise, the reconstructed state is no longer consistent.
  • Evident from the text: the hardware evaluation is limited to a small subset of benchmarks for real devices (24 small circuits on IBM-Q; 4 circuits across vendors), so the empirical results do not cover the full suite.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3550488

Cites

134 of 134 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 2 Not valid · 26 Not checked · 68 Own results and interpretations

29 more claims

Create a free account to see all 107 claims and to verify the 18 references still open.

See all claims — free account
  • Probably not a substantial claim. 64
  • It may state two separate claims as one. 24

Create a free account to see these claims.

Citations

Cites

Show all 134 references

Research works printed but not linked: 1

  1. Juexiao Su, Tianheng Tu, and Lei He. 2016. A quantum annealing approach for Boolean satisfiability problem. In 2016 53rd ACM/EDAC/IEEE Design Automation Conference (DAC). IEEE, 1–6.

Cited by

Items 101–181 of 181

Metadata

Authors
Li, A., Stein, S., Krishnamoorthy, S., & Ang, J.
Affiliations
Pacific Northwest National Laboratory (US)
Venue
ACM Transactions on Quantum Computing
Published
2023-02-24
Volume
4
Issue
2
Pages
1-26
ISSN
  • 2643-6809 (ISSN-L)
  • 2643-6817
OpenAlex
W4288758398
Cited by
183
References
132
Citations by year
2026: 52 · 2025: 74 · 2024: 32 · 2023: 20
FWCI
10.77
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Quantum Computing (publishedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Funders
  • U.S. Department of Energy, Office of Science, National Quantum Information Science Research Centers, Co-design Center for Quantum Advantage (DE-SC0012704)
  • DOE Office of Science User Facility (DE-AC05-00OR22725)
  • Pacific Northwest National Laboratory is operated by Battelle for the U.S. Department of Energy (DE-AC05-76RL01830)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Quantum Computing Algorithms and Architecture
  • Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Quantum Information and Cryptography
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
NISQ devices, OpenQASM, gate density, measurement density, quantum simulation, IBM Quantum, IonQ, quantum machine learning, quantum chemistry

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.