Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

Probably not a substantial claim: may not pass S3 (load-bearing).

Extending to temperature-based sampling would require the rejection-sampling correction of Leviathan et al. [12] and Chen et al. [3].

“Extending to temperature-based sampling would require the rejection-sampling correction of Leviathan et al. [12] and Chen et al. [3]”

• cites Leviathan et al. [12] [12] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML) , 2023. - sourceNot checked yet

  Not checked in the free run.

• cites Chen et al. [3] [3] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 , 2023. - sourceSupported

May combine two claims.

Gemma 4's Per-Layer Embeddings (PLE) are a dense 262K × 8960 matrix-not a standard lookup table.

“Gemma 4's Per-Layer Embeddings (PLE) are a dense 262K × 8960 matrix-not a standard lookup table.”

• cites [5] [5] Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770 , 2026. - sourcePartly supported

  The “Dense and MoE” section says that Gemma 4 E2B and E4B use per-layer embeddings. It does not describe those embeddings as a dense 262K × 8960 matrix or distinguish them from a standard lookup table. The 262,000 figure in “Efficient MTP Decoding” refers to a vocabulary projection for the drafters, not PLE dimensions; no page numbers are available in our copy. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Serving systems built on paged attention support prefix caching through their block-level cache managers.

“Serving systems built on paged attention support prefix caching (reusing the KV cache of an alreadyprocessed prompt prefix instead of re-prefilling it) through their block-level cache managers [11].”

• cites [11] [11] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP) , 2023. - sourceSupported

May combine two claims.

Gemma 4 has layers 15-34 that share K/V projections with earlier layers; the weights are untrained placeholders, and inference reads from source layers' cache via DynamicCache.

“Gemma 4 also exercises a feature Llama doesn't have: layers 15-34 share K/V projections with earlier layers (the weights are untrained placeholders, and inference reads from source layers' cache via DynamicCache ).”

• cites [5] [5] Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770 , 2026. - sourcePartly supported

  The “Dense and MoE” section (page unavailable) says Gemma 4 shares the KV cache in its E2B and E4B models, supporting the general cache-sharing point. It does not identify layers 15–34, say their K/V projection weights are untrained placeholders, or describe inference through DynamicCache; the cited passages also do not establish the comparison with Llama. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recipe-based cache compression such as GEAR targets a different bottleneck: near-lossless 4-bit caches buy capacity (longer contexts or more concurrent streams in fixed memory) at the price of additional decompression compute.

“Recipe-based cache compression such as GEAR [10] targets a different bottleneck: near-lossless 4-bit caches buy capacity (longer contexts or more concurrent streams in fixed memory) at the price of additional decompression compute.”

• cites [10] [10] Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint arXiv:2403.05527 , 2024. - sourceSupported

May combine two claims.

The three WAN methods do not converge on a single number: at L=100 ms/hop the 2-stage sleep-sim reports 11.20 tok/s; the 3-stage queue-proxy reports 2.01; the Tiber DERP run at ∼16 ms relay RTT reports 2.80 uncompressed and 22.88 with top-1 compression.

“at L =100 ms/hop the 2-stage sleep-sim (Table 10) reports 11 . 20 tok/s; the 3-stage queue-proxy at the same hop latency (Table 11) reports 2 . 01; the Tiber DERP run at ∼ 16 ms relay RTT (this section) reports 2 . 80 uncompressed and 22 . 88 with top-1 compression.”

May combine two claims.

K=10 at 50 ms/hop reaches 16.04 tok/s (3.36× over naïve 4.77); at 100 ms/hop it reaches 11.30 tok/s (4.07× over naïve 2.77).

“K =10 at 50 ms/hop reaches 16 . 04 tok/s (3 . 36 × over naïve 4 . 77 at the same hop latency); at 100 ms/hop it reaches 11 . 30 tok/s (4 . 07 × over naïve 2 . 77).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Raw TCP round-trip for 16 KB payloads is 6.5 ms on the 802.11ax LAN.

“Raw TCP round-trip for 16 KB payloads: 6.5 ms on the 802.11ax LAN (measured separately).”

May combine two claims.

The two-node fleet reaches 43.97 tok/s at LAN, which is 1.79× monolithic single-user.

“the two-node fleet reaches 43 . 97 tok/s at LAN (1 . 79 × monolithic single-user)”

May combine two claims.

A CPU-hosted stage 0 works and is measured for Gemma 4 at 9.87 tok/s (CPU→GPU) vs. 10.40 tok/s (GPU→GPU).

“a CPU-hosted stage 0 works and is measured for Gemma 4 (Table 16, CPU → GPU at 9 . 87 tok/s vs. 10 . 40 GPU → GPU)”

May combine two claims.

Intel's network blocks direct UDP between AI PC instances, so all cross-instance traffic is forced through Tailscale's DERP relay servers.

“Intel's network blocks direct UDP between AI PC instances, so all cross-instance traffic is forced through Tailscale's DERP relay servers (Seattle region, ∼ 16 ms RTT to the relay, ∼ 32 ms RTT instance-to-instance via the relay).”

May combine two claims.

Without compression, both streams sit at 1.40 tok/s-well below the 5 tok/s interactive floor; the DERP relay's per-segment queueing makes the full-FP32 logits path unusable.

“Without compression, both streams sit at 1 . 40 tok/s - well below the 5 tok/s interactive floor; the DERP relay's per-segment queueing makes the full-FP32 logits path unusable.”

• AssumptionRests on an assumption: The interactive floor for live inference is 5 tok/s.

May combine two claims.

At 100 ms/hop the naïve path falls below the interactive floor while the full stack remains usable.

“At 100 ms/hop the naïve path falls below the interactive floor while the full stack remains usable.”

• AssumptionRests on an assumption: The interactive floor for live inference is 5 tok/s.

May combine two claims.

Generation length amplifies the effect: acceptance rises from 70.7% to 97.5% and speedup at K=3 climbs from ∼1.3× at 128 tokens to ∼1.6× at 2048 tokens.

“Generation length amplifies the effect: across 128, 512, 1024, and 2048-token runs on the same prompt, acceptance rises from 70 . 7% to 97 . 5% as the model enters its repetition regime, and speedup at K =3 climbs from ∼ 1 . 3 × at 128 tokens to ∼ 1 . 6 × at 2048 tokens.”

May combine two claims.

The composition of multipliers is empirical: 16.33 × 1.797 × 1.499 = 43.99, matching measured 43.97 within 0.05 tok/s.

“Composition is empirical, not a projection: 16 . 33 × 1 . 797 × 1 . 499 = 43 . 99, matching the measured 43 . 97 within 0 . 05 tok/s; all three multipliers were measured in the same paired session”

May combine two claims.

On the 2-stage distributed pipeline, TTFT ranges from 122 ms (short 8-token prompt) to 786 ms (long prompt requiring multi-pass prefill), dominated by prompt-length-dependent compute on stage 0 rather than network overhead.

“On the 2-stage distributed pipeline, TTFT ranges from 122 ms (short 8-token prompt) to 786 ms (long prompt requiring multi-pass prefill), dominated by prompt-length-dependent compute on stage 0 rather than network overhead.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OpenVINO's uniform INT8 KV cache hint ran slower than the default on Arc iGPU.

“the one cache-precision knob we did test-OpenVINO's uniform INT8 KV cache hint-ran slower than the default on Arc iGPU”

May combine two claims.

A four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed with a 70B model, with output token-for-token identical to the same four-node pipeline decoding without speculation.

“a four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed, with output token-for-token identical to the same four-node pipeline decoding without speculation.”

• AssumptionRests on an assumption: 16+ GB of unified memory is not enough to fit a 70B-parameter LLM.

Probably not a substantial claim: may not pass S3 (load-bearing).

All configurations in Table 13 are bit-exact across both/all streams.

“All configurations bit-exact across both/all streams.”

May combine two claims.

The full stack degrades gracefully-per-stream throughput stays at or above the interactive floor through 100 ms/hop-while naïve distributed decode falls below it past ∼25 ms/hop.

“The full stack degrades gracefully-per-stream throughput stays at or above the interactive floor through 100 ms/hop-while naïve distributed decode falls below it past ∼ 25 ms/hop.”

• AssumptionRests on an assumption: The interactive floor for live inference is 5 tok/s.

May combine two claims.

The logits payload returned from the final stage to the coordinator is vocabulary 128,256 × 4 bytes ≈ 501 KB per target verify, which fragments across multiple TCP segments and stretches the realised per-call cost.

“vocabulary 128 , 256 × 4 bytes ≈ 501 KB per target verify, which fragments across multiple TCP segments and stretches the realised per-call cost.”

May combine two claims.

Going from 2-stream to 3-stream at K=5 adds +14.1 tok/s aggregate (from 50.55 to 64.67) at the cost of ∼3.7 tok/s per-user latency, hitting the min(N_users, N_stages) pipelinefill ceiling exactly.

“Going from 2-stream to 3-stream at K =5 adds +14 . 1 tok/s aggregate (from 50 . 55 to 64 . 67) at the cost of ∼ 3 . 7 tok/s per-user latency, hitting the min( N users , N stages ) pipelinefill ceiling exactly.”

May combine two claims.

Provider networks may block UDP outright-Intel Tiber Cloud does.

“provider networks may block UDP outright-Intel Tiber Cloud does (§6.8)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Run-to-run variance on the LAN datapoint is ±0.3 tok/s (∼1%) based on the Table 7 cross-session comparison.

“Run-to-run variance on the LAN datapoint is ± 0 . 3 tok/s ( ∼ 1%) based on the Table 7 cross-session comparison”

Probably not a substantial claim: may not pass S3 (load-bearing).

KV-cached decode sends 16 KB per hop.

“KV-cached decode sends [1, 1, 4096] float32 tensors: 16 KB per hop.”

May combine two claims.

Speculative decoding stacks cleanly with sharding; the spec-decode multiplier is comparable on both paths (1.24× on monolithic target, 1.32× on 3-stage shard target).

“The spec-decode multiplier is comparable on both paths (1 . 24 × on the monolithic target, 1 . 32 × on the 3-stage shard target).”

May combine two claims.

The 3-stage 2-stream stack at 50.55 tok/s exceeds the 2-stage Table 7 headline (43.97); going from 2 to 3 streams adds +14 tok/s aggregate.

“The 3-stage 2-stream stack at 50 . 55 tok/s exceeds the 2-stage Table 7 headline (43 . 97); going from 2 to 3 streams adds +14 tok/s aggregate.”

May combine two claims.

The full stack achieves 43.97 tok/s, 1.79× the monolithic single-user reference, serving two users concurrently.

“for a system-throughput total of 43 . 97 tok/s: 1 . 79 × the monolithic single-user reference on the same hardware, two users served concurrently.”

May combine two claims.

Physical KV trim on Arc B390 iGPU with a 72-token cache costs 45-50 ms per call, and the cost is device-side state invalidation, not numpy work.

“All three variants cluster around 45-50 ms; the cost is device-side state invalidation, not numpy work.”

May combine two claims.

A K=3 speculative step executes one target verify (∼47 ms), 2-3 draft feeds (∼16 ms each), and one correction feed (∼16 ms).

“A K =3 speculative step executes one target verify ( ∼ 47 ms memory-bound forward of K +1=4 tokens), 2-3 draft feeds ( ∼ 16 ms each), and one correction feed ( ∼ 16 ms).”

May combine two claims.

Optimal K shifts from 7 at LAN to 10 at 100 ms/hop.

“Optimal K shifts from 7 at LAN (compute-bound, large drafts waste cycles when compute is cheap) to 10 at 100 ms/hop (network-bound, packing more tokens per round-trip dominates).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Run-to-run variance is typically ±0.3 stddev across sessions.

“Run-to-run variance is typically ± 0 . 3 stddev across sessions.”

May combine two claims.

Greedy spec-decode with top-1 logits compression is bit-exact identical to full-FP32 logits because both sides agree on arg max and the spec-decode acceptance check is a token-id comparison.

“Greedy spec-decode is bit-exact identical to full-FP32 logits because both sides agree on arg max and the spec-decode acceptance check is a token-id comparison.”

May combine two claims.

Output is bit-exact: mono-plus-spec matches mono-only, and shard-plus-spec matches shard-only.

“Output is bit-exact: the mono-plus-spec output matches the mono-only output, and the shard-plus-spec output matches the shard-only output.”

May combine two claims.

XVE occupancy is 60.1% on mono vs. 43.6% on 3-stage.

“measured 60 . 1% on mono vs. 43 . 6% on 3-stage via VTune gpu-hotspots”

May combine two claims.

Under a 100 ms/hop simulated WAN, the two-node configuration stays usable while naïve pipeline-parallel decode falls below the interactive floor.

“under a 100 ms/hop simulated WAN the same configuration stays usable while naïve pipeline-parallel decode falls below the interactive floor.”

May combine two claims.

On real WAN (Tiber Cloud's Tailscale DERP relay), the win is dramatic-8.17× over uncompressed-because every saved 64 KB segment saves a full RTT of relay-server queueing.

“On real WAN (Tiber Cloud's Tailscale DERP relay, §6.8), the win is dramatic - 8 . 17 × over uncompressed - because every saved 64 KB segment saves a full RTT of relay-server queueing.”

May combine two claims.

The 3-stage 2-stream stack hits 50.55 tok/s at K=5, 15% above the 2-stage K=3 headline (43.97); the third stage adds compute parallelism for mbatch without giving up bit-exactness.

“The 3-stage 2-stream stack hits 50 . 55 tok/s at K =5, 15% above the 2-stage K =3 headline (43 . 97); the third stage adds compute parallelism for mbatch (each stream keeps three iGPUs busy) without giving up bit-exactness.”

May combine two claims.

LAN prefers mid K (K=7 wins, K=5 close) where compute dominates and excess draft feeds waste cycles, while 100 ms/hop prefers large K (K=10) where batching enough tokens per round-trip dominates.

“finds that LAN prefers mid K ( K =7 wins, K =5 close) where compute dominates and excess draft feeds waste cycles, while 100 ms/hop prefers large K ( K =10) where batching enough tokens per round-trip dominates.”

May combine two claims.

At 2048-token generation with K=3, cache bloat reaches 1.02× logical length, so no periodic compaction is needed for realistic single-turn lengths.

“At 2048-token generation with K =3, cache bloat reaches 1 . 02 × logical length (most rejected-draft overhead is amortized out by the live-token count), so no periodic compaction is needed for realistic single-turn lengths.”

• AssumptionRests on an assumption: Realistic single-turn lengths are below the model's window such that cache bloat of 1.02× is acceptable.

May combine two claims.

LAN / cloud-adjacent: compute dominates. K=7 peaks at 30.96 tok/s (1.44× over the single-stream LAN baseline of 21.55); K=5 trails closely at 30.42 (1.41×).

“LAN / cloud-adjacent. Compute dominates. K =7 peaks at 30 . 96 tok/s (1 . 44 × over the singlestream LAN baseline of 21 . 55); K =5 trails closely at 30 . 42 (1 . 41 × ).”

Probably not a substantial claim: may not pass S4 (based).

The whole token budget is ∼60 ms.

“every layer removed from the per-token path matters when the whole token budget is ∼ 60 ms.”

May combine two claims.

Llama and Qwen2.5-family shards compile, while Qwen3's exported graph currently does not.

“per-model NPU compiler op support (Llama and Qwen2.5-family shards compile; Qwen3's exported graph currently does not)”

May combine two claims.

The runtime now supports continuous batching on all three device classes, through two mechanisms.

“The runtime now supports it on all three device classes, through two mechanisms.”

May combine two claims.

Aggregate throughput is 16.30 tok/s on v2, 1.57× over single-stream.

“Aggregate throughput is 16 . 30 tok/s on v2, 1 . 57 × over single-stream.”

May combine two claims.

Optimal K shifts up with hop count (K=7 at LAN, K=10 at 100 ms/hop). At 4 DERP-relayed hops per token over Tailscale, K=10 is the empirical winner for 70B 4-stage.

“Optimal K shifts up with hop count (§6.5 sweep: K =7 at LAN, K =10 at 100 ms/hop). At 4 DERP-relayed hops per token over Tailscale, K =10 is the empirical winner for 70B 4-stage.”

May combine two claims.

A monolithic 8B-class model fails to fit on the NPU, while per-stage shards of a 9B model compile and run there.

“(a monolithic 8B-class model fails to fit on the NPU, while per-stage shards of a 9B model compile and run there)”

May combine two claims.

The exported graph builds its 4D additive mask in a single node whose output feeds every attention block, and that mask input already carries a query dimension.

“The exported graph, however, builds its 4D additive mask in a single node whose output feeds every attention block, and that mask input already carries a query dimension.”

May combine two claims.

Per-stream throughput drops from 25.3 to 21.6 tok/s-still 4.3× the 5 tok/s interactive floor.

“Per-stream throughput drops from 25 . 3 to 21 . 6 tok/s - still 4 . 3 × the 5 tok/s interactive floor.”

May combine two claims.

At 4096 tokens the trend reverses (5.00 tok/s, 66.3% accept) as KV cache size starts to dominate per-step compute.

“At 4096 tokens the trend reverses (5 . 00 tok/s, 66 . 3% accept) as KV cache size starts to dominate per-step compute.”

May combine two claims.

On the 3-stage testbed, going from 2 to 3 concurrent streams adds +14.1 tok/s aggregate (from 50.55 to 64.67 tok/s at K=5 LAN), 1.28× scaling against the 2-stream baseline.

“On the 3-stage testbed, going from 2 to 3 concurrent streams adds +14 . 1 tok/s aggregate (from 50 . 55 to 64 . 67 tok/s at K =5 LAN, §6.6), 1 . 28 × scaling against the 2-stream baseline.”

May combine two claims.

The coord process eats ∼20 GB of system RAM during compile (stage 0 + draft, both with 2 InferRequests each), so 32 GB nodes are the minimum.

“The coord process eats ∼ 20 GB of system RAM during compile (stage 0 + draft, both with 2 InferRequests each), so 32 GB nodes are the minimum.”

Classified as probably not a substantial claim.

The batch dimension is pinned to 1.

“The batch dimension is likewise pinned to 1”

May combine two claims.

At 2048 tokens, spec speedup is 1.58×, acceptance is 97.5%, and cache bloat ratio is 1.02×.

“2 , 048 tokens: 1 . 58 × speedup, 97 . 5% acceptance, cache bloat ratio 1 . 02 ×”

May combine two claims.

On the 2-stage distributed pipeline (single-stream, no spec), earlier measurements on an extended-prompt workload showed no per-token degradation across 200 tokens (15.95 tok/s), 1000 tokens (15.55 tok/s), and 10 consecutive prompts (14.46 tok/s aggregate, 0.72 QPS, with 36 ms KV-cache reset between prompts).

“On the 2-stage distributed pipeline (single-stream, no spec), earlier measurements on an extendedprompt workload showed no per-token degradation across 200 tokens (15 . 95 tok/s), 1000 tokens (15 . 55 tok/s), and 10 consecutive prompts (14 . 46 tok/s aggregate, 0 . 72 QPS, with 36 ms KV-cache reset between prompts).”

May combine two claims.

The INFERENCE_PRECISION_HINT='f32' workaround costs -10% on 2-stage and -60% on 1-stage.

“which costs -10% on 2-stage, -60% on 1-stage”

May combine two claims.

The rotary-fixed shards ('v2') compile at default GPU precision and recover full speed.

“The rotary-fixed shards (we call them 'v2') compile at default GPU precision and recover full speed”

May combine two claims.

At 100 ms/hop cross-continental WAN, the fleet-with-speculative-decoding remains interactive (11 tok/s aggregate, 5.6 tok/s per stream) where a naïve distributed pipeline falls to 2.77 tok/s (unusable).

“at 100 ms/hop cross-continental WAN, the fleet-with-speculative-decoding remains interactive (11 tok/s aggregate, 5 . 6 tok/s per stream) where a naïve distributed pipeline falls to 2 . 77 tok/s (unusable).”

May combine two claims.

The 1.66× Gemma 4 E2B micro-batch ratio in Table 19 uses the older external-rotary export (8.12 → 13.51 tok/s); the v2 shards reach 1.57×.

“The 1 . 66 × Gemma 4 E2B micro-batch ratio in this row uses the older external-rotary export (8 . 12 → 13 . 51 tok/s); the v2 shards (Table 16) reach 1 . 57 × .”

May combine two claims.

The gap between scenario (a) and (b) widens with network cost: 1.24-1.44× in-process, 3.11× on the 4-hop relay-mediated WAN, because each verify amortizes the round trips across K+1 positions.

“The gap widens with network cost: 1 . 24-1 . 44 × in-process, 3 . 11 × on the 4-hop relay-mediated WAN, because each verify amortizes the round trips across K +1 positions.”

May combine two claims.

The earlier 2.03× measurement on the original-window 14.51-baseline was right at the structural ceiling; the current measurement window's faster baseline compresses the headroom.

“The earlier 2 . 03 × measurement on the original-window 14.51-baseline was right at the structural ceiling; the current measurement window's faster baseline compresses the headroom.”

May combine two claims.

NPU decode is weight-bound rather than KV-bound: on the Qwen2.5-1.5B INT4 stage measured below, one decode step streams 458 MB of weights against 29.3 MB of KV traffic per token, a 16:1 ratio.

“NPU decode is weight-bound rather than KV-bound: on the Qwen2.5-1.5B INT4 stage measured below, one decode step streams 458 MB of weights against 29.3 MB of KV traffic per token, a 16:1 ratio.”

May combine two claims.

Captured state is compacted at serialization time, keeping only mask-valid positions, so a spec-decoded session is warm-resumable; the draft model, which trails the target by one token, is realigned before the shared suffix feed.

“Captured state is therefore compacted at serialization time, keeping only mask-valid positions, so a spec-decoded session is warm-resumable; the draft model, which trails the target by one token, is realigned before the shared suffix feed.”

May combine two claims.

The cliff is sharp in practice: output stays coherent while the sequence fits the window and breaks the step it crosses.

“The cliff is sharp in practice; output stays coherent while the sequence fits the window and breaks the step it crosses”

Probably not a substantial claim: may not pass S3 (load-bearing).

KV is FP16.

“FP16 KV”

May combine two claims.

FP32 is the only working precision, inflating stage 0 to 7 GB (exceeding Lunar Lake GPU's ∼4 GB allocation limit).

“FP32 is the only working precision, inflating stage 0 to 7 GB (exceeding Lunar Lake GPU's ∼ 4 GB allocation limit).”

May combine two claims.

Reshaping a static shard's batch dimension from 1 to N is rejected at every N tried, on both of the shard graph shapes tested, while the same graphs compile at batch 1.

“Reshaping a static shard's batch dimension from 1 to N is rejected at every N we tried, on both of the shard graph shapes tested, while the same graphs compile at batch 1.”

May combine two claims.

The packed NPU serving mode of §8.6 carries its own, smaller prefix-reuse mechanism, working at the attention-mask level rather than through state capture; its latency savings have been measured and are reported there.

“The packed NPU serving mode of §8.6 carries its own, smaller prefix-reuse mechanism, working at the attention-mask level rather than through state capture; its latency savings have been measured and are reported there.”

May combine two claims.

The cost of per-stream isolation is extra compile time (seconds per additional stream on Arc iGPU) and extra GPU memory (∼2 GB per additional Llama 8B stage per stream).

“The cost is extra compile time (seconds per additional stream on Arc iGPU) and extra GPU memory ( ∼ 2 GB per additional Llama 8B stage per stream)”

May combine two claims.

Long context speeds up further. At 1024 tokens, acceptance rises to 72.2% and throughput climbs to 5.72 tok/s-spec decode benefits from the prompt setting context, opposite the usual per-step compute trend.

“Long context speeds up further. At 1024 tokens, acceptance rises to 72 . 2% and throughput climbs to 5 . 72 tok/s - spec decode benefits from the prompt setting context, opposite the usual per-step compute trend.”

May combine two claims.

Under the three conditions (organization owns AI PCs, data must stay on-premises, latency budget accommodates ∼45-65 ms tokens), a 2-node fleet running the full stack (sharded + micro-batched + speculative) serves two concurrent users of Llama 3.1 8B at 43.97 tok/s aggregate (22.0 tok/s per user) with zero marginal cost and full data locality.

“Under these conditions, a 2-node fleet running the full stack (sharded + micro-batched + speculative) serves two concurrent users of Llama 3.1 8B at 43 . 97 tok/s aggregate (22 . 0 tok/s per user) with zero marginal cost and full data locality.”

May combine two claims.

Going to 2 streams adds +10% (LL coord) or +19% (PL coord) over 1-stream peak.

“Going to 2 streams adds +10% (LL coord) or +19% (PL coord) over 1-stream peak”

May combine two claims.

The Arrow Lake-S Tiber nodes' Xe-LPG iGPU could not handle even the smaller 12-layer stages of the 7-stage paired layout (worker died mid-inference without surfacing a Python exception); the 4 Lunar Lake instances ran the larger 20-layer 4-stage shards cleanly.

“the Arrow Lake-S Tiber nodes' Xe-LPG iGPU could not handle even the smaller 12-layer stages of the 7-stage paired layout (worker died mid-inference without surfacing a Python exception); the 4 Lunar Lake instances ran the larger 20-layer 4-stage shards cleanly.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The OpenVINO NPU compiler rejects both dynamic shapes and ReadValue / Assign state variables, so it cannot compile the stateful dynamic-shape shards of §3.2 at all.

“the OpenVINO NPU compiler rejects both dynamic shapes and ReadValue / Assign state variables, so it cannot compile the stateful dynamic-shape shards of §3.2 at all.”

May combine two claims.

At 1024 tokens, spec speedup is 1.57× and acceptance is 95.1%.

“1 , 024 tokens: 1 . 57 × speedup, 95 . 1% acceptance.”

May combine two claims.

The 1024-token run reaches 5.72 tok/s (+5.5% over the 128-token reference) with acceptance climbing to 72.2%.

“The 1024-token run reaches 5 . 72 tok/s (+5 . 5% over the 128-token reference) with acceptance climbing to 72 . 2%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Static-graph compiles take minutes.

“minutes-long static-graph compiles”

May combine two claims.

The mode serves a whole model on one node; it does not yet compose with the stage chain of §3.3, and the NPU plugin rejects it outright.

“The mode serves a whole model on one node; it does not yet compose with the stage chain of §3.3, and the NPU plugin rejects it outright.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Generated output is coherent; the 70B model elaborates with a follow-up question and answers it correctly, rather than the repetition loop a smaller model often produces.

“Generated output is coherent. first10 stream 0 =”

May combine two claims.

On CPU and GPU, an opt-in serving mode replaces the monolithic LLMPipeline with OpenVINO GenAI's ContinuousBatchingPipeline, which brings paged attention, dynamic split-fuse scheduling, block-level prefix caching, and mid-generation cancellation of individual requests.

“An opt-in serving mode replaces the monolithic LLMPipeline with OpenVINO GenAI's ContinuousBatchingPipeline , which brings paged attention, dynamic split-fuse scheduling, block-level prefix caching, and mid-generation cancellation of individual requests.”

May combine two claims.

The GPU plugin tolerates the empty leading dim; the CPU plugin rejects it.

“The GPU plugin tolerates the empty leading dim; the CPU plugin rejects it.”

Classified as probably not a substantial claim. May combine two claims.

Each slot is an independent request occupying one row of the sequence axis; the batch dimension stays 1 throughout.

“Each slot is an independent request occupying one row of the sequence axis; the batch dimension stays 1 throughout.”

May combine two claims.

On scored tasks the two configurations are identical: 15/20 on short factual questions and 89.5% ordered-atom recall on long-form tasks in both, with identical text throughout.

“On scored tasks the two configurations are identical: 15/20 on short factual questions and 89.5% ordered-atom recall on long-form tasks in both, with identical text throughout”

May combine two claims.

The packed serving mode of §8.6 improves the NPU's multi-user throughput through concurrency, and covers the same 8B-class two-stage pipeline the results tables are built on.

“The packed serving mode of §8.6 improves the NPU's multi-user throughput through concurrency, and covers the same 8B-class two-stage pipeline the results tables are built on”

May combine two claims.

With four requests sharing a 96-token system prompt on the NPU, the later three reach their first token in 0.38 s against 2.26 s for the first, a 5.95× improvement.

“With four requests sharing a 96-token system prompt on the NPU, the later three reach their first token in 0 . 38 s against 2 . 26 s for the first, a 5 . 95 × improvement”

Probably not a substantial claim: may not pass S3 (load-bearing).

The two stages of the Llama 3.1 8B INT4 pipeline compile in about eight minutes together.

“the two stages compile in about eight minutes together”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

WiFi latency varies by ±2 ms (averaging out over many tokens).

“WiFi latency varies by ± 2 ms (averaging out over many tokens).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The ASUS Zenbook S 14 fan runs continuously during benchmarks.

“the ASUS Zenbook S 14 fan runs continuously during benchmarks”

May combine two claims.

A small graph edit replaces the node with a [1,1,S,T] Parameter, handing mask construction to the host; the edit runs on an already-exported stage, with no PyTorch model and no re-trace.

“A small graph edit replaces the node with a [1 , 1 , S, T ] Parameter, handing mask construction to the host; the edit runs on an already-exported stage, with no PyTorch model and no re-trace.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It has not been tried on the 70B-class deployment of §6.11.

“it has not been tried on the 70B-class deployment of §6.11.”

Classified as probably not a substantial claim. May combine two claims.

The slots partition a fixed 1023-position KV window, so per-slot context shrinks as the count rises; exporting a wider static context relaxes the trade.

“The slots partition a fixed 1023-position KV window, so per-slot context shrinks as the count rises; exporting a wider static context relaxes the trade.”

May combine two claims.

Top-1 logits compression (§6.7) gives an 8.17× speedup on DERP-relayed paths but only 1.12× on LAN.

“top-1 logits compression (§6.7) gives an 8 . 17 × speedup on DERP-relayed paths but only 1 . 12 × on LAN.”

Classified as probably not a substantial claim. May combine two claims.

The past-KV window is partitioned into one contiguous region per slot, optionally behind a shared read-only prefix every row may attend to; the last S columns are the current query positions.

“The past-KV window is partitioned into one contiguous region per slot, optionally behind a shared read-only prefix every row may attend to; the last S columns are the current query positions.”

May combine two claims.

The host partitions the fixed KV window into one contiguous region per slot and writes the mask each step.

“The host then partitions the fixed KV window into one contiguous region per slot and writes the mask each step”

May combine two claims.

Sixteen concurrent requests decode at 1.76 ms per token where one request pays 10.57 ms.

“sixteen concurrent requests decode at 1 . 76 ms per token where one request pays 10 . 57 ms”

May combine two claims.

Two of ten 128-token generations differ by one immediately re-converging token, and the 2-stage NPU pipeline lands at the same eight of ten against its own unpacked baseline.

“two of ten 128-token generations differ by one immediately re-converging token, and the 2-stage NPU pipeline lands at the same eight of ten against its own unpacked baseline.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There is no fault tolerance, authentication, or encryption.

“There is no fault tolerance, authentication, or encryption.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A12/20 layer split (vs. the default 16/16) yielded 2% improvement within measurement noise.

“A12/20 layer split (vs. the default 16/16) yielded 2% improvementwithin measurement noise.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All nodes must run identical software stacks (same OpenVINO, same Python).

“All nodes must run identical software stacks (same OpenVINO, same Python).”

May combine two claims.

CPU-hosted stages work (Table 16) but same-SoC CPU stages contend with the GPU for memory bandwidth.

“CPU-hosted stages work (Table 16) but same-SoC CPU stages contend with the GPU for memory bandwidth”

Probably not a substantial claim: may not pass S3 (load-bearing).

We do not report power draw or thermal throttling measurements.

“We do not report power draw or thermal throttling measurements.”

May combine two claims.

On hardware, a 2-stage Llama-3.2-1B INT4 pipeline with both stages compiled to and running on a Lunar Lake NPU, four slots over a 255-position region each, serves four rounds of fifteen sequential then six concurrent requests, 84 in all, with nothing logged on either rank.

“On hardware, a 2-stage Llama-3.2-1B INT4 pipeline with both stages compiled to and running on a Lunar Lake NPU, four slots over a 255-position region each, serves four rounds of fifteen sequential then six concurrent requests, 84 in all, with nothing logged on either rank”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every benchmark reported is scoped to integrated GPUs.

“Every benchmark reported is scoped to integrated GPUs”

May combine two claims.

Each method makes the dominant cost something different-sleep-sim isolates the latency dimension, queue-proxy charges per chunk through a per-byte-delayed forwarder, DERP charges per packet through a real relay server with its own queueing.

“Each method makes the dominant cost something different - sleep-sim isolates the latency dimension, queue-proxy charges per chunk through a per-byte-delayed forwarder, DERP charges per packet through a real relay server with its own queueing.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The mask-based rewind composes with the rejection-sampling correction in principle, but has not been validated empirically.

“the mask-based rewind composes with this correction in principle (the cache state is the same either way), but we have not validated it empirically.”

May combine two claims.

The observed 4.04× at 100 ms/hop falls between the mbatch-alone and full-ceiling bounds, consistent with neither amortization being saturated at this hop latency.

“The observed 4 . 04 × at 100 ms/hop falls between the mbatch-alone and full-ceiling bounds, consistent with neither amortization being saturated at this hop latency.”

May combine two claims.

After both, the network is ∼10% of per-token time on LAN-which bounds what any transport swap can recover at this scale.

“After both, the network is ∼ 10% of per-token time on LAN-which bounds what any transport swap can recover at this scale.”

May combine two claims.

Extending the spec + mbatch coordinator to 3 stages is straightforward and lets us run the full stack on the alpha + charlie + beta testbed.

“Extending the spec + mbatch coordinator to 3 stages is straightforward and lets us run the full stack on the alpha + charlie + beta testbed (§6.1).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

A production transport therefore needs a TCP fallback regardless.

“A production transport therefore needs a TCP fallback regardless.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For multi-segment payloads, QUIC's stream-level loss recovery avoids the cwnd-ramp cost measured in Table 11 and should help materially.

“For multi-segment payloads (the 512 KB FP32 logits return, §6.7), QUIC's stream-level loss recovery avoids the cwnd-ramp cost we measure in Table 11 and should help materially.”

May combine two claims.

The sleep-sim approach is deterministic but lacks real TCP effects like slow-start or jitter; reported speedups should be considered a lower bound on the real WAN advantage.

“Unlike traditional tc/netem WAN emulation, this approach is deterministic but lacks real TCP effects like slow-start or jitter-the reported speedups should be considered a lower bound on the real WAN advantage”

May combine two claims.

On LAN, bandwidth is not the bottleneck; activation compression cannot help.

“On LAN, bandwidth is not the bottleneck-compression cannot help.”

May combine two claims.

Adding hop latency on the 3-stage target-only path costs more wall-clock per token than the sleep-sim suggests, because each token requires four full TCP round-trips with ∼500 KB logits responses-TCP cwnd takes several RTTs to ramp on a cold flow, and persistent connections only partially amortize the warm-cwnd penalty.

“Adding hop latency on the 3-stage target-only path costs more wall-clock per token than the sleep-sim suggests, because each token requires four full TCP round-trips with ∼ 500 KB logits responses - TCP cwnd takes several RTTs to ramp on a cold flow, and persistent connections only partially amortize the warm-cwnd penalty.”

May combine two claims.

Higher K pays for the extra hop because K+1 tokens emitted per target round-trip amortize across 4 network hops instead of 2; the LAN K-sweep of Table 12's target-only path predicted this directional shift, and it now holds in the full stack.

“Higher K pays for the extra hop because K +1 tokens emitted per target round-trip amortize across 4 network hops instead of 2; the LAN K-sweep of Table 12's target-only path predicted this directional shift, and it now holds in the full stack.”

May combine two claims.

The mask-based rewind is correct for greedy because the acceptance check is a simple token-id comparison and the surviving cache is identical regardless of rewind method.

“The mask-based rewind is correct for greedy because the acceptance check is a simple token-id comparison and the surviving cache is identical regardless of rewind method.”

May combine two claims.

On content easier for the draft to predict, speedup climbs toward the theoretical 1+ K · p_accept ceiling; on content where the draft diverges early, it falls to the acceptance floor.

“On content that is easier for the draft to predict (structured output: code, translation, chat boilerplate), the speedup climbs toward the theoretical 1+ K · p accept ceiling; on content where the draft diverges early, it falls to the acceptance floor.”

• AssumptionRests on an assumption: The theoretical speedup ceiling for speculative decoding is 1 + K · p_accept.

May combine two claims.

Each additional stage adds a network hop per token but also adds an iGPU that microbatching can keep busy.

“Each additional stage adds a network hop per token but also adds an iGPU that microbatching can keep busy.”

May combine two claims.

Both regimes hit roughly the same multiplier through different mechanisms because at high latency the network term dominates the per-token wall time and the multipliers approach each other.

“both regimes hit roughly the same multiplier through different mechanisms because at high latency the network term dominates the per-token wall time and the multipliers approach each other.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every table row in the paper is backed by a committed benchmark script and raw log under the reproduction/ directory.

“every table row below is backed by a committed benchmark script and raw log under the reproduction/ directory in the companion repository”

May combine two claims.

For greedy decoding the coordinator only needs arg max + the top probability for spec-decode bookkeeping; sending only the top-1 token id and probability (∼8 bytes) preserves correctness.

“For greedy decoding the coordinator only needs arg max + the top probability for spec-decode bookkeeping; sending only the top-1 token id and probability ( ∼ 8 bytes) preserves correctness.”

May combine two claims.

The gap to the sleep-sim 2-stage figures comes from the topology change (2 hops/token vs. 1) and the absence of speculative decoding-not from the proxy method itself.

“The gap to the sleep-sim 2-stage figures comes from the topology change (2 hops/token vs. 1) and the absence of speculative decoding-not from the proxy method itself.”

May combine two claims.

For measured K=3 with p_accept=0.707, the spec-only ceiling is 3.12× over the spec-less single-stream baseline; combined with the mbatch ≈2×, the full-stack ceiling against naïve is ≈6.2×.

“For our measured K =3 with p accept =0 . 707, the spec-only ceiling is 3 . 12 × over the spec-less single-stream baseline; combined with the mbatch ≈ 2 × , the full-stack ceiling against naïve is ≈ 6 . 2 × .”

• AssumptionRests on an assumption: The theoretical speedup ceiling for speculative decoding is 1 + K · p_accept.

May combine two claims.

On wired gigabit Ethernet the same 16 KB round trip is sub-millisecond, removing ∼10% of the per-token budget.

“on wired gigabit Ethernet the same 16 KB round trip is sub-millisecond, removing ∼ 10% of the per-token budget (Table 8).”

The quote only partly states this claim.

Mask-based KV rewind makes speculative decoding pay off on stateful OpenVINO models.

“Second, we make speculative decoding pay off on stateful OpenVINO models”

• AssumptionRests on an assumption: Canonical speculative decoding implementations rely on paged attention or dedicated rewind APIs that are not available on stateful OpenVINO models.

May combine two claims.

Masking out low-utility positions yields the same attention result at no device cost, so the machinery built here would carry cache-reduction policies like SkipKV.

“Masking those positions out instead yields the same attention result at no device cost, so the machinery built here would carry such policies as well.”

• cites SkipKV [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The sweet spot is moderate K (5-7) that balances acceptance with draft cost; precise winner shifts within that band across runs by ∼3%.

“The sweet spot is moderate K (5-7) that balances acceptance with draft cost; precise winner shifts within that band across runs by ∼ 3%.”

Classified as probably not a substantial claim.

All results in this paper use greedy decoding (arg max token selection).

“All results in this paper use greedy decoding (arg max token selection).”

Probably not a substantial claim: may not pass S4 (based).

A stateful OpenVINO shard offers no such API: its KV cache is opaque per-InferRequest device state.

“A stateful OpenVINO shard offers no such API: its KV cache is opaque perInferRequest device state.”

May combine two claims.

An NPU shard must be both static-shape and stateless.

“An NPU shard must be both static-shape and stateless.”

May combine two claims.

The cost profile works out because capture runs once per generation rather than once per decode step; the ∼48 ms state round-trip of Table 4 that was fatal for per-step rewind is negligible next to a full prefill.

“The cost profile works out because capture runs once per generation rather than once per decode step; the ∼ 48 ms state round-trip of Table 4 that was fatal for per-step rewind is negligible next to a full prefill.”

May combine two claims.

The slightly lower ratio on the new exports is consistent: both numerator and denominator move up, but the single-stream baseline moves up by more, leaving less stage-idle time for the second stream to absorb.

“The slightly lower ratio on the new exports is consistent: both numerator (single-stream) and denominator (2-stream aggregate) move up, but the single-stream baseline moves up by more, leaving less stage-idle time for the second stream to absorb.”

May combine two claims.

Under greedy decoding, speculative decoding is bit-exact-(b) produces the same tokens as (a), only faster.

“Under greedy decoding, speculative decoding is bit-exact-(b) produces the same tokens as (a), only faster”

May combine two claims.

Smaller per-stage layer counts did not save the Arrow Lake instance from failure, so 'finer splits as a workaround' cannot be recommended.

“The takeaway is asymmetric: smaller per-stage layer counts did not save the Arrow Lake instance from failure, so we cannot recommend 'finer splits as a workaround.'”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Output is byte-correct on every config.

“Output byte-correct on every config.”

Probably not a substantial claim: may not pass S2 (checkable).

Bit-exact equivalence to the 4-stage target-only path on the same shards is the strongest correctness check currently available.

“Bit-exact equivalence to the 4-stage target-only path on the same shards is the strongest correctness check we have for now”

May combine two claims.

The NPU export path pins every input dimension (batch 1, decode sequence length 1, a context window fixed at export time) and skips the make-stateful transformation, so the KV cache becomes explicit graph ports: each shard is a pure function (input, attention mask, position ids, past KV) → (output, present KV).

“The NPU export path pins every input dimension (batch 1, decode sequence length 1, a context window fixed at export time) and skips the make-stateful transformation, so the KV cache becomes explicit graph ports: each shard is a pure function (input , attention mask , position ids , past KV) → (output , present KV).”

May combine two claims.

The two graphs can compile to different devices, prefilling on one and decoding on another.

“The two graphs can even compile to different devices, prefilling on one and decoding on another.”

May combine two claims.

An NPU stage frees the iGPU for other work at lower power.

“an NPU stage frees the iGPU for other work at lower power.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

A 2-stage pipeline has an inherent inefficiency for single requests: while stage 1 computes, stage 0 sits idle. With balanced 25 ms stages, utilization is roughly 50%.

“A 2-stage pipeline has an inherent inefficiency for single requests: while stage 1 computes, stage 0 sits idle. With balanced 25 ms stages, utilization is roughly 50%.”

May combine two claims.

A mixed-precision export path (FP32 embedding, INT4 decoder layers) would solve this but requires NNCF to support per-subgraph precision control.

“A mixed-precision export path (FP32 embedding, INT4 decoder layers) would solve this but requires NNCF to support per-subgraph precision control.”

May combine two claims.

The 70B result is the strongest validation of the paper's central thesis: a fleet of consumer Intel AI PCs can run a model that would not fit on any of them individually, at interactive throughput, with bit-exact output relative to the same topology's target-only path.

“The 70B result is the strongest validation of the paper's central thesis: a fleet of consumer Intel AI PCs can run a model that would not fit on any of them individually, at interactive throughput, with bit-exact output relative to the same topology's target-only path.”

May combine two claims.

The multi-stage protocol, in which each stage captures and restores only its own layers, runs end-to-end on fleet hardware but has not yet received the same byte-exact certification.

“The multi-stage protocol, in which each stage captures and restores only its own layers, runs end-to-end on fleet hardware but has not yet received the same byte-exact certification.”

May combine two claims.

Spec decode amortization grows with deeper pipelines. A K=10 verify amortizes ∼7.5 emitted tokens per target round-trip; for a 4-hop topology each emitted token pays ∼13% of the round-trip latency it would on a K=1 naïve path.

“Spec decode amortization grows with deeper pipelines. A K =10 verify amortizes ∼ 7 . 5 emitted tokens per target round-trip; for a 4-hop topology each emitted token pays ∼ 13% of the round-trip latency it would on a K =1 naïve path.”

Probably not a substantial claim: may not pass S4 (based).

Each OpenVINO InferRequest should, per the Model API documentation, maintain independent KV cache state.

“Each OpenVINO InferRequest should , per the Model API documentation, maintain independent KV cache state.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The v2 export removed a precisionhint penalty that afflicts any compile attempted on OV 2026.1+.

“the v2 export having removed a precisionhint penalty that afflicts any compile attempted on OV 2026.1+.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Continuous batching is the more fluid discipline used by paged-attention servers: requests join a running batch as they arrive and leave it as they finish.

“Continuous batching [19] is the more fluid discipline used by paged-attention servers: requests join a running batch as they arrive and leave it as they finish.”

• cites [19] [19] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2022. -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The 70B-class regime where the model genuinely cannot fit on one node is the regime the distributed pipeline exists for.

“The 70B-class regime where the model genuinely cannot fit on one node - is the regime the distributed pipeline exists for.”

Probably not a substantial claim: may not pass S4 (based).

Continuous batching, in which requests join and leave a running batch as they arrive and finish, lifts that restriction.

“Continuous batching, in which requests join and leave a running batch as they arrive and finish, lifts that restriction (§8.6).”

May combine two claims.

The 4-stage topology imposes 3 worker round-trips per spec verify (stage 0 runs in-coord; stages 1-3 are remote), and these per-call DERP costs amortize across the 1+0.653·10 ≈ 7.5 tokens emitted per call when K=10.

“the 4-stage topology imposes 3 worker round-trips per spec verify (stage 0 runs in-coord; stages 1-3 are remote), and these per-call DERP costs amortize across the 1+0 . 653 · 10 ≈ 7 . 5 tokens emitted per call when K =10.”

May combine two claims.

The per-stream throughput drops below the 5 tok/s interactive floor, but for batch-style multi-user serving the aggregate 6.43 tok/s is the relevant metric.

“the per-stream throughput drops below the 5 tok/s interactive floor, but for batch-style multi-user serving the aggregate 6 . 43 tok/s is the relevant metric.”

Probably not a substantial claim: may not pass S4 (based).

Llama 3.1 70B INT4 (∼36 GB) is the smallest model in the 70B class that fits a 4-node fleet at 32 GB system RAM each.

“Llama 3.1 70B INT4 ( ∼ 36 GB) is the smallest model in the 70B class that fits a 4-node fleet at 32 GB system RAM each.”

May combine two claims.

Chat templates that rewrite earlier turns destroy prefix stability, and such prompts miss and fall back to a cold prefill, as does every failure path (missing capture on any stage, model-fingerprint mismatch, restore timeout).

“Chat templates that rewrite earlier turns (models that re-render thinking blocks, for example) destroy prefix stability, and such prompts miss and fall back to a cold prefill, as does every failure path (missing capture on any stage, model-fingerprint mismatch, restore timeout).”

May combine two claims.

Output between the two concurrent streams is byte-identical (same prompt, deterministic generation), confirming that the cross-stream KV isolation is correct.

“Output between the two concurrent streams is byte-identical (same prompt, deterministic generation), confirming that the cross-stream KV isolation is correct.”

May combine two claims.

Per-stream throughput at 1-stream (5.42) and PL-coord 2-stream (3.21) brackets the 5 tok/s interactive floor: a 70B fleet running the full stack is at the boundary of usable interactive chat for one user and trades that floor for ∼6 tok/s aggregate across two users.

“Per-stream throughput at 1-stream (5 . 42) and PL-coord 2-stream (3 . 21) brackets the 5 tok/s interactive floor: a 70B fleet running the full stack is at the boundary of usable interactive chat for one user and trades that floor for ∼ 6 tok/s aggregate across two users.”

May combine two claims.

Because the serialized state is opaque to everything but the producing shard, reuse is whole-sequence: a captured sequence accelerates any later prompt it prefixes, which covers session resume but not block-level sharing of partial prefixes between unrelated requests.

“Because the serialized state is opaque to everything but the producing shard, reuse is whole-sequence: a captured sequence accelerates any later prompt it prefixes, which covers session resume but not block-level sharing of partial prefixes between unrelated requests.”

May combine two claims.

The exporter can emit a second static graph with the sequence dimension pinned to a chunk width C instead of 1, sharing the same host KV ring byte-identically, so the prompt is consumed C tokens per forward and only decode runs at sequence length 1.

“Prefill does not have to feed token-by-token: the exporter can emit a second static graph with the sequence dimension pinned to a chunk width C instead of 1, sharing the same host KV ring byte-identically, so the prompt is consumed C tokens per forward and only decode runs at sequence length 1.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multi-stream runs are bit-exact across both streams.

“Multi-stream runs are bit-exact across both streams.”

May combine two claims.

The mask-based rewind leaves rejected-draft K/V in physical cache, but rejected-draft overhead is per-step constant while the live-token count grows linearly, so the bloat ratio converges to 1.

“the mask-based rewind leaves rejected-draft K/V in physical cache, but rejected-draft overhead is per-step constant while the live-token count grows linearly-bloat ratio converges to 1”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This is deliberately smaller machinery than the capture-and-restore caching of §8.4: one entry, populated by whichever request arrives first, no eviction, and no block-level sharing between partially-overlapping prompts.

“This is deliberately smaller machinery than the capture-and-restore caching of §8.4: one entry, populated by whichever request arrives first, no eviction, and no block-level sharing between partially-overlapping prompts.”

May combine two claims.

The slot count is baked into the IR shape, so changing it means re-exporting and restarting every stage.

“the slot count is baked into the IR shape, so changing it means re-exporting and restarting every stage”

May combine two claims.

The 4-slot Llama-3.2-1B worker admits four requests at once where the unpacked path serializes them.

“4-slot Llama-3.2-1B worker admits four requests at once where the unpacked path serializes them”

Classified as probably not a substantial claim.

The headline sweep in Table 10 uses time.sleep() on the worker's recv/send boundaries to inject deterministic one-way delay.

“The headline sweep in Table 10 uses time.sleep() on the worker's recv/send boundaries to inject deterministic one-way delay”

May combine two claims.

The placement solver never chose the NPU because single-stream decode there costs about 4× the iGPU per token; packing divides that cost by the concurrent slot count.

“The placement solver never chose the NPU because single-stream decode there costs about 4 × the iGPU per token; packing divides that cost by the concurrent slot count.”

May combine two claims.

The three composing techniques each address a distinct bottleneck and are individually validated with bit-exact output vs. their respective baselines.

“The three composing techniques each address a distinct bottleneck and are individually validated with bit-exact output vs. their respective baselines:”

May combine two claims.

The bottleneck for asymmetric shard planning is network round-trip cost, not compute imbalance between stages.

“The bottleneck is network round-trip cost, not compute imbalance between stages.”

May combine two claims.

Reuse is single-stage only, unlike the packing itself: the shared region is populated at admission, which only stage 0 performs, so a downstream stage would open no shared columns for the tokens stage 0 skipped prefilling.

“Reuse is single-stage only, unlike the packing itself: the shared region is populated at admission, which only stage 0 performs, so a downstream stage would open no shared columns for the tokens stage 0 skipped prefilling.”

May combine two claims.