CoChat
Sign Up
PreprintCited by 29Quantum Computing Algorithms and ArchitectureQuantum Information and CryptographyQuantum many-body systems

Benchmarking Quantum Processor Performance at Scale

Open access
McKay, David C., Hincks, Ian, Pritchett, Emily J., Carroll, Malcolm, Govia, Luke C. G., Merkel, Seth T.·2023·arXiv
DOI 10.48550/arxiv.2311.05933arXiv 2311.05933
Open this paper in CoChat — free accountOpen PDF

Abstract

As quantum processors grow, new performance benchmarks are required to capture the full quality of the devices at scale. While quantum volume is an excellent benchmark, it focuses on the highest quality subset of the device and so is unable to indicate the average performance over a large number of connected qubits. Furthermore, it is a discrete pass/fail and so is not reflective of continuous improvements in hardware nor does it provide quantitative direction to large-scale algorithms. For example, there may be value in error mitigated Hamiltonian simulation at scale with devices unable to pass strict quantum volume tests. Here we discuss a scalable benchmark which measures the fidelity of a connecting set of two-qubit gates over $N$ qubits by measuring gate errors using simultaneous direct randomized benchmarking in disjoint layers. Our layer fidelity can be easily related to algorithmic run time, via $γ$ defined in Ref.\cite{berg2022probabilistic} that can be used to estimate the number of circuits required for error mitigation. The protocol is efficient and obtains all the pair rates in the layered structure. Compared to regular (isolated) RB this approach is sensitive to crosstalk. As an example we measure a $N=80~(100)$ qubit layer fidelity on a 127 qubit fixed-coupling "Eagle" processor (ibm\_sherbrooke) of 0.26(0.19) and on the 133 qubit tunable-coupling "Heron" processor (ibm\_montecarlo) of 0.61(0.26). This can easily be expressed as a layer size independent quantity, error per layered gate (EPLG), which is here $1.7\times10^{-2}(1.7\times10^{-2})$ for ibm\_sherbrooke and $6.2\times10^{-3}(1.2\times10^{-2})$ for ibm\_montecarlo.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.05933

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • State-of-the-art many-qubit platforms have only recently demonstrated entangling gates with 99.9% fidelity, far short of the <10−10 error rates needed for many applications.

    “State-of-the-art many-qubit platforms have only recently demonstrated entangling gates with 99.9% fidelity9,10, far short of the <10−10 error rates needed for many applications11,12.”

    From Quantum error correction below the surface code threshold (2025) · cites this paper as [9]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. E. v. d. Berg, Z. K. Minev, A. Kandala, and K. Temme, Probabilistic error cancellation with sparse pauli-lindblad models on noisy quantum processors, arXiv preprint arXiv:2201.09866 (2022). · arXiv
  2. M. Amico, H. Zhang, P. Jurcevic, L. Bishop, P. Na- tion, A. Wack, and D. C. McKay, Defining stan- dard strategies for quantum benchmarks, arXiv preprint arXiv:2303.02108 (2023). · arXiv
  3. A. Wack, H. Paik, A. Javadi-Abhari, P. Jurcevic, I. Faro, J. M. Gambetta, and B. R. Johnson, Quality, speed, and scale: three key attributes to measure the perfor- mance of near-term quantum computers, arXiv preprint arXiv:2110.14108 (2021). · arXiv
  4. E. Magesan, J. M. Gambetta, and J. Emerson, Charac- terizing quantum gates via randomized benchmarking, Phys. Rev. A 85, 042311 (2012). · DOI · Paper page
  5. J. Helsen, I. Roth, E. Onorati, A. Werner, and J. Eisert, General framework for randomized benchmarking, PRX Quantum 3, 020357 (2022). · DOI
  6. J. M. Gambetta, A. D. Córcoles, S. T. Merkel, B. R. Johnson, J. A. Smolin, J. M. Chow, C. A. Ryan, C. Rigetti, S. Poletto, T. A. Ohki, M. B. Ketchen, and M. Steffen, Characterization of addressability by simulta- neous randomized benchmarking, Phys. Rev. Lett. 109, 240504 (2012). · DOI
  7. T. Lubinski, S. Johri, P. Varosy, J. Coleman, L. Zhao, J. Necaise, C. H. Baldwin, K. Mayer, and T. Proctor, Application-oriented performance benchmarks for quan- tum computing, arXiv preprint arXiv:2110.03137 (2021). · arXiv
  8. K. Mesman, Z. Al-Ars, and M. Möller, Qpack: Quan- tum approximate optimization algorithms as univer- sal benchmark for quantum computers, arXiv preprint arXiv:2103.17193 (2021). · arXiv
  9. J. R. Finžgar, P. Ross, L. Hölscher, J. Klepsch, and A. Luckow, Quark: A framework for quantum comput- ing application benchmarking, in 2022 IEEE Interna- tional Conference on Quantum Computing and Engineer- ing (QCE) (IEEE, 2022) pp. 226-237.
  10. T. Tomesh, P. Gokhale, V. Omole, G. S. Ravi, K. N. Smith, J. Viszlai, X.-C. Wu, N. Hardavellas, M. R. Martonosi, and F. T. Chong, Supermarq: A scalable quantum benchmark suite, in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) (IEEE, 2022) pp. 587-603.
  11. V. Zhang and P. D. Nation, Characterizing quantum processors using discrete time crystals, arXiv preprint arXiv:2301.07625 (2023). · arXiv
  12. A. Kurlej, S. Alterman, and K. M. Obenland, Bench- marking and analysis of noisy intermediate-scale trapped ion quantum computing architectures, in 2022 IEEE In- ternational Conference on Quantum Computing and En- gineering (QCE) (IEEE, 2022) pp. 247-258.
  13. T. Lubinski, C. Coffrin, C. McGeoch, P. Sathe, J. Apanavicius, and D. E. B. Neira, Optimization ap- plications as quantum performance benchmarks, arXiv preprint arXiv:2302.02278 (2023). · arXiv
  14. M. Kordzanganeh, M. Buchberger, M. Povolotskii, W. Fischer, A. Kurkin, W. Somogyi, A. Sagingalieva, M. Pflitsch, and A. Melnikov, Benchmarking simulated and physical quantum processing units using quantum and hybrid algorithms, arXiv preprint arXiv:2211.15631 (2022). · arXiv
  15. P. S. Mundada, A. Barbosa, S. Maity, T. Stace, T. Merkh, F. Nielson, A. R. Carvalho, M. Hush, M. J. Biercuk, and Y. Baum, Experimental benchmarking of an automated deterministic error suppression workflow for quantum algorithms, arXiv preprint arXiv:2209.06864 (2022). · arXiv
  16. A. Li, S. Stein, S. Krishnamoorthy, and J. Ang, Qasm- bench: A low-level quantum benchmark suite for nisq evaluation and simulation, ACM Transactions on Quan- tum Computing 10.1145/3550488 (2022). · DOI · Paper page
  17. A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and J. M. Gambetta, Validating quantum computers us- ing randomized model circuits, Physical Review A 100, 032328 (2019).
  18. S. Boixo, S. V. Isakov, V. N. Smelyanskiy, R. Babbush, N. Ding, Z. Jiang, M. J. Bremner, J. M. Martinis, and H. Neven, Characterizing quantum supremacy in near- term devices, Nature Physics 14, 595 (2018). · DOI · Paper page
  19. T. Proctor, S. Seritan, K. Rudinger, E. Nielsen, R. Blume-Kohout, and K. Young, Scalable randomized benchmarking of quantum computers using mirror cir- cuits, Phys. Rev. Lett. 129, 150502 (2022). · DOI · Paper page
  20. J. Hines, D. Hothem, R. Blume-Kohout, B. Whaley, and T. Proctor, Fully scalable randomized benchmarking without motion reversal (2023), arXiv:2309.05147 [quant- ph]. · arXiv
  21. C. H. Baldwin, K. Mayer, N. C. Brown, C. Ryan- Anderson, and D. Hayes, Re-examining the quantum vol- ume test: Ideal distributions, compiler optimizations, confidence intervals, and scalable resource estimations, Quantum 6, 707 (2022). · DOI · Paper page
  22. X. Xu, S. Benjamin, J. Sun, X. Yuan, and P. Zhang, A herculean task: Classical simulation of quantum comput- ers (2023). · DOI
  23. Y. Liu, Y. Chen, C. Guo, J. Song, X. Shi, L. Gan, W. Wu, W. Wu, H. Fu, X. Liu, D. Chen, G. Yang, and J. Gao, Validating quantum-supremacy experiments with exact and fast tensor network contraction (2022). · DOI
  24. A. M. Dalzell, S. McArdle, M. Berta, P. Bienias, C.- F. Chen, A. Gilyén, C. T. Hann, M. J. Kastoryano, E. T. Khabiboulline, A. Kubica, G. Salton, S. Wang, and F. G. S. L. Brandão, Quantum algorithms: A sur- vey of applications and end-to-end complexities (2023), arXiv:2310.03011 [quant-ph]. · arXiv
  25. M. Liepelt, T. Peduzzi, and J. R. Wootton, En- hanced repetition codes for the cross-platform com- parison of progress towards fault-tolerance (2023), arXiv:2308.08909 [quant-ph]. · arXiv
  26. S. Ferracin, A. Hashim, J.-L. Ville, R. Naik, A. Carignan- Dugas, H. Qassim, A. Morvan, D. I. Santiago, I. Siddiqi, and J. J. Wallman, Efficiently improving the performance of noisy quantum computers (2022), arXiv:2201.10672 [quant-ph]. · arXiv
  27. T. J. Proctor, A. Carignan-Dugas, K. Rudinger, E. Nielsen, R. Blume-Kohout, and K. Young, Direct randomized benchmarking for multiqubit devices, Phys. Rev. Lett. 123, 030503 (2019). · DOI · Paper page
  28. A. Morvan and et. al, Phase transition in random circuit sampling (2023), arXiv:2304.11119 [quant-ph]. · arXiv
  29. A. Erhard, J. J. Wallman, L. Postler, M. Meth, R. Stricker, E. A. Martinez, P. Schindler, T. Monz, J. Emerson, and R. Blatt, Characterizing large-scale quantum computers via cycle benchmarking, Nat. Comm. 10, 5347 (2019). · DOI
  30. J. Helsen, X. Xue, L. M. K. Vandersypen, and S. Wehner, A new class of efficient randomized benchmarking proto- cols, npj Quantum Inf 5, 71 (2019). · DOI
  31. S. Kimmel, M. P. da Silva, C. A. Ryan, B. R. Johnson, and T. Ohki, Robust extraction of tomographic infor- mation via randomized benchmarking, Phys. Rev. X 4, 011050 (2014). · DOI
  32. D. C. McKay, A. W. Cross, C. J. Wood, and J. M. Gam- betta, Correlated randomized benchmarking (2020). · DOI
  33. R. Harper, S. Flammia, and J. Wallman, Efficient learn- ing of quantum noise, Nat. Phys. 16, 1184 (2020). · DOI
  34. P. D. Nation and M. Treinish, Suppressing quantum circuit errors due to system variability, arXiv preprint arXiv:2209.15512 (2022). · arXiv
  35. P. Murali, J. M. Baker, A. Javadi-Abhari, F. T. Chong, and M. Martonosi, Noise-adaptive compiler mappings for noisy intermediate-scale quantum computers, in Proceed- ings of the twenty-fourth international conference on ar- chitectural support for programming languages and oper- ating systems (2019) pp. 1015-1029.
  36. J. Emerson, R. Alicki, and K. Życzkowski, Scalable noise estimation with random unitary operators, Journal of Optics B: Quantum and Semiclassical Optics 7, S347 (2005). · DOI · Paper page
  37. L. H. Pedersen, N. M. Møller, and K. Mølmer, Fidelity of quantum operations, Physics Letters A 367, 47 (2007). · DOI
  38. K. X. Wei, E. Magesan, I. Lauer, S. Srinivasan, D. F. Bogorin, S. Carnevale, G. A. Keefe, Y. Kim, D. Klaus, W. Landers, N. Sundaresan, C. Wang, E. J. Zhang, M. Steffen, O. E. Dial, D. C. McKay, and A. Kan- dala, Hamiltonian engineering with multicolor drives for fast entangling gates and quantum crosstalk cancellation, Phys. Rev. Lett. 129, 060501 (2022). · DOI
  39. B. K. Mitchell, R. K. Naik, A. Morvan, A. Hashim, J. M. Kreikebaum, B. Marinelli, W. Lavrijsen, K. Nowrouzi, D. I. Santiago, and I. Siddiqi, Hardware-efficient microwave-activated tunable coupling between supercon- ducting qubits, Phys. Rev. Lett. 127, 200502 (2021). · DOI
  40. D. Greenbaum, Introduction to quantum gate set tomog- raphy (2015), arXiv:1509.02921 [quant-ph]. · arXiv
  41. M. A. Nielsen, A simple formula for the average gate fi- delity of a quantum dynamical operation, Physics Letters A 303, 249 (2002). Appendix A: Additional Data · DOI · Paper page

Cited by

Sorted by citations

Authors
McKay, D. C., Hincks, I., Pritchett, E. J., Carroll, M., Govia, L. C. G., & Merkel, S. T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2311.05933
arXiv
2311.05933
OpenAlex
W4388650951
Cited by
29
References
0
Citations by year
2026: 4 · 2025: 12 · 2024: 12 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Army Research Office (W911NF-21-1-, W911NF-21-1-0002, W911NF)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Quantum Computing Algorithms and Architecture
  • Physical Sciences › Computer Science › Artificial Intelligence › Quantum Information and Cryptography
  • Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Quantum many-body systems
Keywords
crosstalk, error mitigation, Hamiltonian simulation, quantum volume, scalable quantum computing, IBM Heron

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review