The DOI matches its Crossref record; 134 of the 134 references it cites were found in a registry or an index; 11 of the 51 sources its claims cite were checked for support.
QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation
Open accessAbstract
The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and so on, trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, comprising 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench .
Overview
CoChat Summary
## TL;DR
QASMBench is an OpenQASM-based benchmark suite for evaluating and simulating NISQ quantum devices, compilers, and simulators using real quantum workloads from many domains. The paper also introduces four circuit metrics and reports fidelity experiments on 12 IBM-Q machines plus cross-vendor comparisons on IonQ and Rigetti systems.
## Key findings
- QASMBench contains benchmark circuits organized into small (2–5 qubits), medium (6–15 qubits), and large (>15 qubits) categories; the largest circuit is `adder_n127` with 3991 gates and 127 qubits.
- The authors propose four circuit metrics: gate density, retention lifespan, measurement density, and entanglement variance, in addition to width and depth, to characterize execution efficiency and NISQ sensitivity.
- In their IBM-Q evaluation, they ran 24 circuits from the small category on 12 IBM-Q machines, using density matrix state tomography, for 25,000 circuit evaluations total and 8192 shots per evaluation.
- They report that fidelity is generally higher for shallower circuits, and that larger machines do not necessarily give higher fidelity; for example, the paper says shallow circuits generally have higher fidelity than deep ones, and the 127-qubit IBM-Q Washington showed reduced fidelity compared with smaller machines.
- In cross-vendor tests on 4 circuits (Deutsch, DNN, Grover, iSWAP), IonQ’s trapped-ion QPU generally had higher fidelity for relatively shallow circuits, but performance dropped for deeper circuits; the paper attributes this to IonQ’s 99.35% single-gate fidelity versus 99.90% for IBM-Q Washington.
## Methods
- The paper presents a benchmark suite built around OpenQASM circuits, drawing routines from multiple domains including chemistry, simulation, linear algebra, search, optimization, arithmetic, machine learning, fault tolerance, cryptography, and quantum communication.
- It categorizes benchmarks by qubit count into small, medium, and large groups and reports circuit properties such as qubits, gate counts, and CX counts for each benchmark.
- For evaluation, it uses fidelity defined via the distance between the ideal state from a noiseless classical simulation and the reconstructed noisy state from density matrix tomography; it also compares results across IBM-Q, IonQ, and Rigetti devices.
## Limitations
- The paper states that density matrix tomography is extremely computationally expensive and requires 3^n circuit evaluations for an n-qubit circuit, making it feasible only for small circuits.
- It also states that tomography must be performed within a short calibration window; otherwise, the reconstructed state is no longer consistent.
- Evident from the text: the hardware evaluation is limited to a small subset of benchmarks for real devices (24 small circuits on IBM-Q; 4 circuits across vendors), so the empirical results do not cover the full suite.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 2 Not valid · 26 Not checked · 68 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The qec_dist3 benchmark performing a 3-distance 5-qubit surface code which is developed based on [90].”
What the sources say
- Partly supportedcites [90] Kristel Michielsen, Madita Nocon, Dennis Willsch, Fengping Jin, Thomas Lippert, and Hans De Raedt. 2017. Benchmarking gate-based quantum computers. Computer Physics Communications 220 (2017), 44ś55. sourceExperimental — accuracy not yet measured
Section 6.2 ('Distance-three 5-qubit code') confirms that the paper implements a 3-distance, 5-qubit code ([[5,1,3]]) with an encoding circuit run on the IBM-QE (~33 µs), so a benchmark built on this work does perform a distance-three 5-qubit code. However, our copy does not call that code a surface code: Section 6 ('Error correction') explicitly contrasts the two five-qubit codes studied — 'The first is a particular distance-two surface code already analyzed by Devitt' and 'The second is an instance of the perfect [[5, 1, 3]] code' — and Section 6.1 labels the distance-two [[5,1,2]] code as the surface code, while Section 6.2 describes the [[5,1,3]] code as a perfect quantum error-correcting code. The 'surface code' descriptor in the citing sentence is therefore not supported by this source. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“it actually leverages the noise presented in the NISQ devices [30]”
What the sources say
- Supportedcites [30] Andrew W Cross, Graeme Smith, and John A Smolin. 2015. Quantum learning robust against noise. Physical Review A 92, 1 (2015), 012327. source
The abstract (p. 1) states that noise 'can be an unlikely ally' and that 'the presence of noise is essential for creating this quantum-classical separation', and the Introduction (p. 1) says the quantum advantage 'occurs because of the noise' for the parity/LPN setting. The Conclusion (p. 4) ties this to noisy near-term hardware: quantum supremacy for learning 'should be possible ... using several hundred noisy qubits, i.e. without the use of quantum error-correction', and the Introduction notes 'the physical implementation of any oracle on bare qubits will inevitably be noisy'. Our copy never uses the acronym 'NISQ' — the paper frames the leverage as belonging to the quantum parity learner (classical LPN being the intractable side) — but the substance of the paraphrase is stated. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“it is computationally intractable for classical approaches [101]”
What the sources say
- Partly supportedcites [101] Daniel K Park, June-Koo K Rhee, and Soonchil Lee. 2018. Noise-tolerant parity learning with one quantum bit. Physical Review A 97, 3 (2018), 032327. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) states that 'The LPN problem is thought to be computationally intractable classically [5-8]', and Sec. II.A (p. 2) adds that 'The best-known classical algorithms for LPN has superpolynomial complexity in n [5-8]' — so the source clearly supports the classical-hardness point. However, the paper hedges the intractability as a widely held belief and, as fact, only asserts superpolynomial complexity of the best *known* classical algorithms (no proven lower bound), whereas the citing sentence prints it flatly as 'it is computationally intractable for classical approaches'. The dropped hedge is the part not supported as stated. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that shallow circuits in general exhibit higher idelity than deep circuits, likely due to less decay impact from a shorter execution time”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Applications in QASMBench can be launched and veriied on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum.”
- Also stated in: § Introduction · plain
“Circuits from QASMBench can be directly uploaded and evaluated in IBM-Q machines as well as other platforms (e.g., Rigetti through the Qiskit interface, IonQ and Quantinuum through Microsoft Azure Quantum [92]).”
What the sources say
- Not matchedcites [92] Microsoft. [n.d.]. Azure Quantum. URL: https://azure.microsoft.com/en-us/services/quantum/.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
29 more claims
Create a free account to see all 107 claims and to verify the 18 references still open.
- Probably not a substantial claim. 64
- It may state two separate claims as one. 24
Citations
Cites
Quantum Computation and Quantum Information
Nielsen, M. A., Chuang, I., & Grover, L. K. (2002) · American Journal of Physics · doi:10.1119/1.1463744 · Cited by 21,895
On the Einstein Podolsky Rosen paradox
Bell, J. S. (1964) · Physics Physique Fizika · doi:10.1103/physicsphysiquefizika.1.195 · Cited by 12,252
Quantum Computing in the NISQ era and beyond
Preskill, J. (2018) · Quantum · doi:10.22331/q-2018-08-06-79 · Cited by 8,830
A fast quantum mechanical algorithm for database search
Grover, L. K. (1996) · Proceedings of the twenty-eighth annual ACM symposium on Theory of computing - STOC '96 · doi:10.1145/237814.237866 · Cited by 8,830
Algorithms for quantum computation: discrete logarithms and factoring
Shor, P. W. (2002) · Proceedings 35th Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.1994.365700 · Cited by 8,713
Gisin, N., Ribordy, G., Tittel, W., & Zbinden, H. (2002) · Reviews of Modern Physics · doi:10.1103/revmodphys.74.145 · Cited by 8,515
Quantum cryptography: Public key distribution and coin tossing
Bennett, C. H., & Brassard, G. (2014) · Theoretical Computer Science · doi:10.1016/j.tcs.2014.05.025 · Cited by 8,299
Quantum cryptography: Public key distribution and coin tossing
Bennett, C. H., & Brassard, G. (2020) · arXiv · doi:10.48550/arxiv.2003.06557 · Cited by 8,299
Quantum supremacy using a programmable superconducting processor
Arute, F., Arya, K., Babbush, R., Bacon, D., Bardin, J. C., Barends, R., Biswas, R., Boixo, S., Brandao, F. G. S. L., Buell, D. A., et al. (2019) · Nature · doi:10.1038/s41586-019-1666-5 · Cited by 7,720
Simulating physics with computers
Feynman, R. P. (1982) · International Journal of Theoretical Physics · doi:10.1007/bf02650179 · Cited by 7,691
Research works printed but not linked: 1
- Juexiao Su, Tianheng Tu, and Lei He. 2016. A quantum annealing approach for Boolean satisfiability problem. In 2016 53rd ACM/EDAC/IEEE Design Automation Conference (DAC). IEEE, 1–6.
Cited by
MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing
Quetschlich, N., Burgholzer, L., & Wille, R. (2023) · Quantum · doi:10.22331/q-2023-07-20-1062 · Cited by 202
SupermarQ: A Scalable Quantum Benchmark Suite
Tomesh, T., Gokhale, P., Omole, V., Ravi, G. S., Smith, K. N., Viszlai, J., Wu, X.-C., Hardavellas, N., Martonosi, M. R., & Chong, F. T. (2022) · arXiv · doi:10.48550/arxiv.2202.11045 · Cited by 143
“QASMBench is a collection of benchmark circuits that range in size from two to a thousand qubits [30].”
BackgroundA Hardware-Aware Heuristic for the Qubit Mapping Problem in the NISQ Era
Niu, S., Suau, A., Staffelbach, G., & Todri-Sanial, A. (2020) · IEEE Transactions on Quantum Engineering · doi:10.1109/tqe.2020.3026544 · Cited by 103
Quantum Software Components and Platforms: Overview and Quality Assessment
Serrano, M., Cruz-Lemus, J. A., Pérez‐Castillo, R., & Piattini, M. G. (2022) · ACM Computing Surveys · doi:10.1145/3548679 · Cited by 129
Optimization Applications as Quantum Performance Benchmarks
Lubinski, T., Coffrin, C., McGeoch, C., Sathe, P., Apanavicius, J., & Neira, D. E. B. (2023) · arXiv · doi:10.48550/arxiv.2302.02278 · Cited by 62
Metadata
- Authors
- Li, A., Stein, S., Krishnamoorthy, S., & Ang, J.
- Affiliations
- Pacific Northwest National Laboratory (US)
- Venue
- ACM Transactions on Quantum Computing
- Published
- 2023-02-24
- Volume
- 4
- Issue
- 2
- Pages
- 1-26
- ISSN
- 2643-6809 (ISSN-L)
- 2643-6817
- OpenAlex
- W4288758398
- Cited by
- 183
- References
- 132
- Citations by year
- 2026: 52 · 2025: 74 · 2024: 32 · 2023: 20
- FWCI
- 10.77
- Type
- journal article
- Language
- en
- Open access
- green
- www.osti.gov
- Locations
- ACM Transactions on Quantum Computing (publishedVersion)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- U.S. Department of Energy, Office of Science, National Quantum Information Science Research Centers, Co-design Center for Quantum Advantage (DE-SC0012704)
- DOE Office of Science User Facility (DE-AC05-00OR22725)
- Pacific Northwest National Laboratory is operated by Battelle for the U.S. Department of Energy (DE-AC05-76RL01830)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Quantum Computing Algorithms and Architecture
- Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Quantum Information and Cryptography
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- NISQ devices, OpenQASM, gate density, measurement density, quantum simulation, IBM Quantum, IonQ, quantum machine learning, quantum chemistry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.