First Experiences in Performance Benchmarking with the New SPEChpc 2021 Suites
Open accessAbstract
Modern High Performance Computing (HPC) sys-tems are built with innovative system architectures and novel programming models to further push the speed limit of computing. The increased complexity poses challenges for performance portability and performance evaluation. The Standard Perfor-mance Evaluation Corporation (SPEC) has a long history of producing industry-standard benchmarks for modern computer systems. SPEC's newly released SPEChpc 2021 benchmark suites, developed by the High Performance Group, are a bold attempt to provide a fair and objective benchmarking tool designed for state-of-the-art HPC systems. With the support of multiple host and accelerator programming models, the suites are portable across both homogeneous and heterogeneous architectures. Different workloads are developed to fit system sizes ranging from a few compute nodes to a few hundred compute nodes. In this work we present our first experiences in performance benchmarking the new SPEChpc2021 suites and evaluate their portability and basic performance characteristics on various popular and emerging HPC architectures, including x86 CPU, NVIDIA GPU, and AMD GPU. This study provides a first-hand experience of executing the SPEChpc 2021 suites at scale on production HPC systems, discusses real-world use cases, and serves as an initial guideline for using the benchmark suites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Benchmarking is a critical process in computer science, serving as a standardized method to evaluate and compare the performance of hardware and software systems.
“Benchmarking is a critical process in computer science, serving as a standardized method to evaluate and compare the performance of hardware and software systems [15].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [15]
Cites
Bailey, D. H., Barszcz, E., Barton, J. T., et al. (1991) · The International Journal of High Performance Computing Applications · doi:10.1177/109434209100500306 · Cited by 2,405 · see citing works
The LINPACK Benchmark: past, present and future
Dongarra, J. J., Luszczek, P., & Petitet, A. (2003) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.728 · Cited by 813 · see citing works
The Scalable Heterogeneous Computing (SHOC) benchmark suite
Danalis, A., Marin, G., McCurdy, C., et al. (2010) · doi:10.1145/1735688.1735702 · Cited by 674 · see citing works
Improving performance via mini-applications.
Crozier, P., Thornquist, H., Numrich, R. W., Williams, A. W., Edwards, H., Keiter, E. R., Rajan, M., Willenbring, J., Doerfler, D. W., & Heroux, M. A. (2009) · doi:10.2172/993908 · Cited by 448 · see citing works
Knüpfer, A., Rössel, C., Mey, D. A., et al. (2012) · Springer eBooks · doi:10.1007/978-3-642-31476-6_7 · Cited by 388 · see citing works
Bucek, J., Lange, K.-D., & von Kistowski, J. (2018) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3185768.3185771 · Cited by 363 · see citing works
Spradling, C. D. (2007) · ACM SIGARCH Computer Architecture News · doi:10.1145/1241601.1241625 · Cited by 157 · see citing works
Performance characteristics of the multi-zone NAS parallel benchmarks
Jin, H. H., & der Wijngaart, R. F. V. (2006) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2005.06.016 · Cited by 97 · see citing works
A microbenchmark suite for OpenMP 2.0
Bull, J. M., & ONeill, D. H. (2001) · ACM SIGARCH Computer Architecture News · doi:10.1145/563647.563656 · Cited by 92 · see citing works
SPEC OMP2012 — An Application Benchmark Suite for Parallel Systems Using OpenMP
Müller, M., Baron, J., Brantley, W. C., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30961-8_17 · Cited by 79 · see citing works
SPEC ACCEL: A Standard Application Suite for Measuring Hardware Accelerator Performance
Juckeland, G., Brantley, W. C., Chandrasekaran, S., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-17248-4_3 · Cited by 66 · see citing works
Self-citationHPGMG 1.0: A Benchmark for Ranking High Performance Computing Systems
Adams, M. F., Brown, J., Shalf, J. M., Straalen, B. V., Strohmaier, E., & Williams, S. (2014) · doi:10.2172/1131029 · Cited by 62 · see citing works
SPEC MPI2007—an application benchmark suite for parallel systems using MPI
Müller, M., van Waveren, M., Lieberman, R., et al. (2009) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.1535 · Cited by 60 · see citing works
HPCG Benchmark Technical Specification
Sandia National Laboratories (SNL-NM), A., Heroux, M. A., Administration, U. A. S. F. H. R. A., et al. (2013) · doi:10.2172/1113870 · Cited by 43 · see citing works
Large System Performance of SPEC OMP2001 Benchmarks
SPEC High-Performance Group, Saito, H., Gaertner, G., et al. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47847-7_34 · Cited by 34 · see citing works
MPI runtime error detection with MUST: Advances in deadlock detection
Hilbrich, T., Protze, J., Schulz, M., et al. (2012) · 2012 International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1109/sc.2012.79 · Cited by 34 · see citing works
Benchmarking with real industrial applications: the SPEC High-Performance Group
Eigenmann, R., & Hassanzadeh, S. (1996) · IEEE Computational Science and Engineering · doi:10.1109/99.486757 · Cited by 31 · see citing works
Wu, X., & Taylor, V. (2011) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/1964218.1964228 · Cited by 28 · see citing works
Juckeland, G., Hernández, Ó., Jacob, A. C., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46079-6_33 · Cited by 19 · see citing works
Self-citationlo2s — Multi-core System and Application Performance Analysis for Linux
Ilsche, T., Schöne, R., Bielert, M., et al. (2017) · Proceedings · doi:10.1109/cluster.2017.116 · Cited by 14 · see citing works
Evaluating Performance Portability of Accelerator Programming Models using SPEC ACCEL 1.2 Benchmarks
Boehm, S., Pophale, S., Vergara, V. M., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02465-9_51 · Cited by 14 · see citing works
Self-citationScore-P and OMPT: Navigating the Perils of Callback-Driven Parallel Runtime Introspection
Feld, C., Convent, S., Hermanns, M.-A., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28596-8_2 · Cited by 12 · see citing works
Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs
Afzal, A., Hager, G., & Wellein, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50743-5_20 · Cited by 11 · see citing works
Performance Analysis for Target Devices with the OpenMP Tools Interface
Cramer, T., Dietrich, R. A., Terboven, C., et al. (2015) · doi:10.1109/ipdpsw.2015.27 · Cited by 8 · see citing works
SPEC HPC2002: The Next High-Performance Computer Benchmark
SPEC High-Performance Group, Eigenmann, R., Gaertner, G., et al. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47847-7_3 · Cited by 5 · see citing works
Proxy App Prospectus for ECP Application Development Projects
Cook, J., (NNSA), U. N. N. S. A., Finkel, H., et al. (2017) · doi:10.2172/1477829 · Cited by 4 · see citing works
Comparing High Performance Computing Accelerator Programming Models
Pophale, S., Boehm, S., & Vergara, V. M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34356-9_14 · Cited by 2 · see citing works
Self-citationEvaluating the Performance of OpenMP Offloading on the NEC SX-Aurora TSUBASA Vector Engine
Cramer, T., Kosmynin, B., Moll, S., et al. (2021) · Supercomputing Frontiers and Innovations · doi:10.14529/jsfi210204 · Cited by 2 · see citing works
First Experiences in Performance Benchmarking with the New SPEChpc 2021 Suites - Measurement Data
Brunst, H., Chandrasekaran, S., Ciorba, F. M., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5573119 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
- Authors
- Brunst, H., Chandrasekaran, S., Ciorba, F. M., Hagerty, N., Henschel, R., Juckeland, G., Li, J., Vergara, V. G. M., Wienke, S., & Zavala, M.
- Affiliations
- Technische Universität Dresden (DE)
- University of Delaware (US)
- University of Basel (CH)
- Oak Ridge National Laboratory (US)
- Indiana University Bloomington (US)
- Helmholtz-Zentrum Dresden-Rossendorf (DE)
- Texas Advanced Computing Center (US)
- Oak Ridge National Laboratory,Oak Ridge,TN,USA
- RWTH Aachen University (DE)
- ORCID
- Venue
- 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
- Publisher
- IEEE
- Published
- 2022-05
- Pages
- 675-684
- OpenAlex
- W4286307885
- Cited by
- 22
- References
- 29
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 8 · 2023: 6 · 2022: 1
- FWCI
- 3.82
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- RWTH Publications (RWTH Aachen) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (1814609)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- performance benchmarking, high-performance computing, performance portability, heterogeneous architectures, GPU computing, NVIDIA GPUs, AMD GPUs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
