Cited by 5Parallel Computing and Optimization TechniquesComputer Graphics and Visualization TechniquesDistributed and Parallel Computing Systems

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; none of the 39 sources its claims cite is checked for support yet.

Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems

Open access

Li, Ruihao, Hanindhito, Bagus, Yadav, Sanjana, Wu, Qinzhe, Kavi, Krishna, Mehta, Gayatri, Yadwadkar, Neeraja J., John, Lizy K.·2025·ACM Transactions on Architecture and Code Optimization

DOI 10.1145/3746231

Open this paper in CoChat — free account Open PDF

Abstract

Driven by the increasing demands of machine learning, heterogeneous systems combining CPUs and GPUs have emerged as the dominant architecture for parallel computing in recent years. To optimize memory management and data transfer between CPUs and GPUs, Nvidia GPUs have introduced unified virtual memory ( UVM ) and pinned memory ( PM ) over the last decade. UVM can avoid explicit memory copies and potentially overlap GPU kernel computations with CPU-GPU data transfer. PM ensures that data with high locality remains in the main memory, preventing it from being paged out. In addition to these two techniques, asynchronous memory copy ( Async Memcpy ) was introduced recently in Nvidia GPUs to improve the CPU-GPU pipeline further. By utilizing Async Memcpy , the data transfer from GPU global memory to shared memory can be overlapped with GPU computations, adding an additional stage to the CPU-GPU data transfer pipeline. A thorough performance analysis of how Async Memcpy affects the current UVM and PM CPU-GPU data transfer scheme is desired. In this article, we provide performance implications of the combined effect of UVM , PM , and Async Memcpy , exploring which applications benefit from which combination of these features. We implement all these features on a suite of 25 workloads, including microbenchmarks and realworld applications. We observe an average performance gain of 24% when utilizing UVM and a 34% gain when employing PM on realworld applications, compared to not applying any data transfer optimization techniques. The performance benefits of Async Memcpy vary across different workloads. For workloads featuring extensive shared memory usage and high compute density (e.g., kmeans and lud ), Async Memcpy delivers around a 20% performance improvement over using UVM or PM alone. In other workloads like knn , we note a 20% performance degradation when using Async Memcpy . Furthermore, we conduct an in-depth investigation of the GPU kernel using performance counters to uncover the root causes of performance differences among various data transfer models. We also perform sensitivity analyses to examine how the number of blocks and threads, as well as the L1-cache/shared memory partitioning, impact performance. We explore future research directions aimed at enhancing the data transfer pipeline by overlapping memory allocation with data transfer and computation across GPU kernels.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper characterizes how pinned memory (PM), unified virtual memory (UVM), and asynchronous memory copy (Async Memcpy) affect CPU-GPU data-transfer pipelines across 25 workloads and four machine configurations. It finds that UVM/PM benefits are highly machine-dependent, Async Memcpy is workload-dependent, and combining them helps only when each technique helps individually.

Key findings

UVM and PM are sensitive to machine system configuration. On Machine-C, UVM improved microbenchmark performance by 48.13% and 50.53% over baseline for Super and Mega inputs, while PM improved it by 57.82% and 56.63%; on Machine-B, UVM caused 93.43% and 84.67% slowdowns, and PM improved performance by only 16.99% and 18.56%. For real-world non-DL applications, UVM improved performance by 23.99% and 4.61% on Machine-C and Machine-D, while PM improved it by 33.90% and 22.80%; on Machine-B, UVM caused a 41.68% slowdown.
Async Memcpy alone had little average overall effect: microbenchmarks slowed by 2.05% and 1.72% on Machine-C for Super and Mega inputs, and by 6.12% and 4.91% on Machine-D; real-world non-DL applications showed only 0.35% slowdown and 0.43% speedup on Machine-C and Machine-D. Workload-specific effects were large: vector_seq saw 42.42% GPU kernel time reduction but only 7.37% overall improvement, 3DCONV saw 143.37% kernel time increase, lud improved 20.03%, and knn slowed 25.22% on Machine-D.
Combining Async Memcpy with UVM/PM can help when each helps individually. For Mega input on Machine-C, uvm_async gave 49.21% improvement over baseline versus 50.53% for uvm, and pm_async gave 55.28% versus 56.63% for pm; however, vector_seq and vector_rand with uvm_async achieved 63.50% and 63.37% time savings over baseline, better than uvm’s 59.98% and 61.25%.
For DL workloads, PM did not improve inference or training, causing 196.43% and 25.30% slowdowns versus baseline, though it reduced CPU-GPU transfer time by 29.75% for inference and 49.14% for training. Async Memcpy had minimal effect on DL workloads, while UVM benefited yolov3 inference because CPU-GPU transfer exceeded 70% of total execution time.

Methods

The study uses four machines: Machine-A (Intel Skylake, A100), Machine-B (AMD Milan, A100), Machine-C (Intel Sapphire Rapids, A100), and Machine-D (AMD Genoa, H100). Six configurations are compared: baseline, async, uvm, uvm_async, pm, and pm_async.
The benchmark suite contains 25 workloads: 7 microbenchmarks and 18 real-world applications (10 non-DL and 8 DL, with training and inference counted separately). Each workload was run 30 times; main experiments focus on Machine-C and Machine-D with Super and Mega input sizes.
Performance counters were collected with Nvidia Nsight Compute, Nsight Systems, and CUPTI. The paper also performs sensitivity studies on CUDA block count, threads per block, and L1-cache/shared-memory partition, and prototypes an inter-job pipeline that overlaps data allocation/deallocation with GPU kernel execution.

Limitations

The paper states that its prototype for overlapping data allocation with kernel execution can cause out-of-memory errors for Mega input sizes because memory allocated by the running kernel is not released before allocating memory for the next kernel, doubling GPU memory capacity requirements. It also states that for tiny input sizes, the overhead of initializing a separate CUDA stream for data allocation exceeds the CUDA kernel cost.
The paper states that PM can introduce overhead from frequent allocation/deallocation, especially for DL frameworks that allocate PM per tensor or per layer, and that UVM benefits depend heavily on machine system configuration and are best assessed through runtime profiling.
Evident from the text: the characterization focuses on single-GPU scenarios and selected benchmark suites, and the paper notes that no automatic compiler tools exist for converting programs to PM, UVM, or Async Memcpy versions, so programmers must hand-tune CUDA programs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3746231

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 39 cited sources checked · 0 supported · 0 flagged

A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed performance gains of 24% and 34% with UVM and PM, respectively, on realworld applications.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- The authors state this as an assumption: Unless otherwise specified, UVM is used with a prefetcher in this article.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance effect of UVM on different machines does not have strong correlations with the number of page faults.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
Load-bearingStrength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- The authors state this as an assumption: Unless otherwise specified, UVM is used with a prefetcher in this article.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance of UVM and PM is also sensitive to the type of machine system configurations (same as the case of microbenmarks).”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
Load-bearingStrength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- The authors state this as an assumption: Unless otherwise specified, UVM is used with a prefetcher in this article.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The performance of a workload can benefit from using a combination of Async Memcpy and UVM/PM if it benefits from using each technique individually.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
Load-bearingStrength:plain· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Async Memcpy can be used together with either PM or UVM.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, the ease of programming provided by UVM does not always guarantee a performance improvement [4,6,48].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 4.1 Microbenchmarks
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [4]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [6]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [48]
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“UVM [31] creates a shared memory address space between CPU hosts and GPU devices, allowing both to access the same virtual memory.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [31]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
In the paper
“In memory-bound applications, data transfer times between the CPU and GPU can significantly surpass the time required for GPU kernel execution, by as much as 50 times [16].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [16]
Load-bearingStrength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
In the paper
“Async Memcpy complicates the data transfer pipeline because additional GPU resources are needed to coordinate the transfer from global memory to shared memory along with SM computation [51].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [51]
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“PM ensures that the memory allocated on the (CPU) host is mapped into the GPU address space, preventing it from being swapped out of main memory to storage devices [14,30,41].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [14]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [30]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [41]
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In PM, additional system calls are required to guarantee pinned pages physically reside in the main memory of the system [14,30].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [14]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [30]
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“If the allocation PM regions with high locality, the CPU-GPU data transfer time can be reduced significantly by utilizing DMA data transfer [19].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 4.1 Microbenchmarks
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [19]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Breaking down the execution time provides developers with practical guidelines: workloads with both high compute density and heavy shared memory demands can benefit from Async Memcpy, UVM is advantageous for workloads with regular data access patterns, and PM offers benefits for workloads with less intensive data allocation.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- The authors state this as an assumption: Unless otherwise specified, UVM is used with a prefetcher in this article.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, in an online inference serving system with a warm start [2,24,27,57], the data allocation overhead can be amortized.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 4.3 RealWorld Deep Learning Applications
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [2]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [24]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [27]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [57]
Strength:plain· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“Our prior analysis proofed that prefetcher can improve the UVM performance[26].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [26]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes

It may state two separate claims as one. 70
The quote only partly states this claim. 1
Probably not a substantial claim. 36
Borderline: one of the tests for a substantial claim is unclear. 1

In the paper
“This feature enables data to be loaded directly from global memory into the shared memory of streaming multiprocessors (SMs).”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [36]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“While AMD's Infinity Fabric technology remains unchanged from Rome to Milan [29], Genoa leverages Infinity Fabric 3.0 with GMI3 interface support for more stable and predictable cross-chiplet data transfers [10].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 3.3 Benchmark and Machine Configurations
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [29]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [10]
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For example, Svedin et al. [51] compared A100 performance with four previous generations of GPUs, and in their experiments on A100, they observed up to 1.25× performance improvement from Async Memcpy.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 2.2 Asynchronous Memcpy
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [51]
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“While there is a wealth of prior studies of PM [14,30,41], UVM [5,6,22,59], and Async Memcpy [51] separately, the performance implications of combined effects of these mechanisms for a given workload remain unclear.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [14]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [30]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [41]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [5]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [6]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [22]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [59]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [51]
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In UVM, page faults can happen on the GPU side when the accessed data is not in the page table, which blocks the data transfer and downgrades the overall system performance [5,6].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [5]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [6]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“While large inputs are commonly used in benchmarks to reduce measurement noise, prior work [26] shows that bigger inputs do not always yield stable performance in CPU-GPU heterogeneous systems.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 3.3 Benchmark and Machine Configurations
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [26]
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
In the paper
“In this case, the data allocation for the second job (kernel) can occur while the first job (kernel) is being processed, which could be leveraged in Kernel-as-a-Service (Kaas) systems [42].”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 8.2 Directions for New Data Transfer Models
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [42]
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“An enhanced UVM version supports prefetch data from global memory to L2 cache[5,22], reducing the global memory to shared memory time.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [5]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [22]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“We also compared the performance of our own implementation with cutlass[33] to guarantee the efficacy of our kernel implementations.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 9 Conclusion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [33]
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Consequently, the primary overhead stems from PM allocation [1,7,45], which distinguishes these applications from microbenchmarks and other non-DL real-world applications.”
§ Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems > 4.3 RealWorld Deep Learning Applications
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [1]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [7]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [45]
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.

98 of 98

The paper relies on this without support.
DL frameworks typically allocate PM on a per-tensor, per-layer basis, resulting in a large number of allocation function calls.
“The reason is that DL frameworks typically allocate PM on a per-tensor, per-layer basis, resulting in a large number of allocation function calls.”
The paper relies on this without support.
During the training phase, convolutional deep neural networks depend heavily on gemm kernels.
“During the training phase, convolutional deep neural networks depend heavily on gemm (general matrix-to-matrix multiplication) kernels.”
The paper relies on this without support.
PM and UVM cannot be used together.
“However, PM and UVM cannot be used together.”
The paper relies on this without support.
Async Memcpy can be used together with either PM or UVM.
“Async Memcpy can be used together with either PM or UVM.”
The authors state this as an assumption.
Unless otherwise specified, UVM is used with a prefetcher in this article.
“If not specified, we assume UVM is used with prefetcher in this article.”

5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Baseline execution-time shares exceed the total
Major
The stated baseline transfer and GPU-kernel shares total 99.80% of execution time. The separately stated baseline allocation share of 18.99% cannot also be part of that same total.
Two different Mega-size UVM improvements on Machine-C
Minor
For the same Machine-C comparison of UVM against baseline, the Mega-size improvement is reported as both 50.53% and 51.53%.
PM-only result is attributed to async-only
Minor
The 56.63% Mega-size improvement is previously identified as the PM-only result. The async-only configuration is instead reported as a 1.72% slowdown on Machine-C.
The stated large matrix input size differs from Table 3
Minor
The prototype calls a 4K×4K matrix the large input size used in Figure 3, but Table 3 assigns 8K * 8K to the Large 2D-grid category; 4K * 4K is its Medium category.

4 of 4

LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Rodinia: A benchmark suite for heterogeneous computing
Che, S., Boyer, M. J., Meng, J., Tarjan, D., Sheaffer, J. W., Lee, S.-H., & Kevin Skadron, Y. (2009) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc.2009.5306797 · Cited by 3,100
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., et al. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 · doi:10.1145/3620665.3640366 · Cited by 737
The Llama 3 Herd of Models
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
NVIDIA A100 Tensor Core GPU: Performance and Innovation
Choquette, J., Gandhi, W., Giroux, O., Stam, N., & Krashinsky, R. (2021) · IEEE Micro · doi:10.1109/mm.2021.3061394 · Cited by 410
Where is the data? Why you cannot debate CPU vs. GPU performance without the answer
Gregg, C., & Hazelwood, K. (2011) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2011.5762730 · Cited by 292
Untitled work
(2021) · doi:10.1145/3458817.3476205 · Cited by 257

Flagged first, then by citations

Show all 30 references

Research works printed but not linked: 9

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. \(\lbrace\) TensorFlow \(\rbrace\) : A system for \(\lbrace\) Large-Scale \(\rbrace\) machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283.
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming \(\lbrace\) throughput-latency \(\rbrace\) tradeoff in \(\lbrace\) LLM \(\rbrace\) inference with \(\lbrace\) sarathi-serve \(\rbrace\) . In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117–134.
10.5555/2015039.2015535 · DOI
Guyue Huang Yang Bai Liu Liu Yuke Wang Bei Yu Yufei Ding and Yuan Xie. 2023. Alcop: Automatic load-compute pipelining in deep learning compiler for ai-gpus. Proceedings of Machine Learning and Systems 5 (2023) 680–694.
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. \(\lbrace\) AlpaServe \(\rbrace\) : Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 663–679.
Dan Negrut, Radu Serban, Ang Li, and Andrew Seidl. 2014. Unified memory in cuda 6.0. a brief overview of related data access and transfer issues. SBEL, Madison, WI, USA, Technical Reports TR-2014-09 (2014).
SB Shriram, Anshuj Garg, and Purushottam Kulkarni. 2019. Dynamic memory management for gpu-based training of deep neural networks. In Proceedings of the 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 200–209.
Yuke Wang, Boyuan Feng, Zheng Wang, Tong Geng, Kevin Barker, Ang Li, and Yufei Ding. 2023. MGG: Accelerating graph neural networks with fine-grained intra-kernel communication-computation pipelining on multi-GPU platforms. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 779–795.
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for \(\lbrace\) transformer-based \(\rbrace\) generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538.

Sorted by citations

Evaluating the benefits of Argument Fusion: Optimizing host-device communication
Sotiropoulos, D.-O., Capotondi, A., & Marongiu, A. (2026) · ACM International Conference on Computing Frontiers: Workshops and Special Sessions (CF Companion) · doi:10.1145/3801488.3806378 · Cited by 2
БАГАТОКРИТЕРІАЛЬНА ОПТИМІЗАЦІЯ ПАМ’ЯТІ В ЗАДАЧАХ МАТЕМАТИЧНОГО МОДЕЛЮВАННЯ ДЛЯ ГІБРИДНИХ КОМП’ЮТЕРНИХ АРХІТЕКТУР
Popov, A. V., & Pavliuk, A. (2025) · Математичне моделювання · doi:10.31319/2519-8106.2(53)2025.342456
High-Throughput sCMOS Image Real-Time Compression System with FPGA-GPU Heterogeneous Co-processing
Zhang, D., Xiao, Y., Yang, X., & Jiang, T. (2026) · International Conference on Information Science, Electrical and Automation Engineering (ISEAE) · doi:10.1109/iseae69422.2026.11544303
Ottimizzare i trasferimenti dati nei sistemi eterogenei: una visione unificata di tecniche di fusione e separazione statiche e dinamiche
Marongiu, A., Capotondi, A., & Sotiropoulos, D.-O. (2026) · IRIS UNIMORE (University of Modena and Reggio Emilia)
When and Why to Use TMA: A Workload-Grounded Characterization of Tensor Memory Acceleration on NVIDIA Hopper GPUs
Bose, C. D., Avalos, C., Pan, J., & Rogers, T. (2026) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc71020.2026.00039

See all 5

Authors: Li, R., Hanindhito, B., Yadav, S., Wu, Q., Kavi, K., Mehta, G., Yadwadkar, N. J., & John, L. K.
Affiliations: The University of Texas at Austin (US)
University of North Texas (US)
ORCID: Li, Ruihao
Hanindhito, Bagus
Yadav, Sanjana
Wu, Qinzhe
Kavi, Krishna
Mehta, Gayatri
Yadwadkar, Neeraja J.
John, Lizy K.
Venue: ACM Transactions on Architecture and Code Optimization
Publisher: Association for Computing Machinery (ACM)
Published: 2025-09-19
Volume: 22
Issue: 3
Pages: 1-26
ISSN: 1544-3566 (ISSN-L)
1544-3973
DOI: 10.1145/3746231
OpenAlex: W4411670522
Cited by: 5
References: 30
Citations by year: 2026: 4 · 2025: 1
FWCI: 2.49
Type: journal article
Language: en
Open access: diamond
doi.org
Licence: cc-by
Locations: ACM Transactions on Architecture and Code Optimization (publishedVersion, cc-by)
Funders: NSF (#2326894 and #2425655)
Topics: Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
Physical Sciences › Computer Science › Computer Networks and Communications › Distributed and Parallel Computing Systems
Keywords: CPU-GPU data transfer, pipelining, heterogeneous computing, memory management, performance counters, memory allocation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks