DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Review date: 2026-07-08 Review author: Zhongzhu Zhou Paper reviewed: DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation Paper authors: Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang arXiv: 2607.05147 Status: Preprint (July 2026 — Peking University, DeepSeek-AI)

Short Answer

Large parallel speculative-decoding drafters can propose many tokens in a single forward pass, but they collapse into “acceptance decay” toward the end of the block because they predict every position independently. DSpark fixes this with a semi-autoregressive drafter — a heavy parallel backbone that stays fast, plus a cheap sequential correction head that injects just enough intra-block dependency to stop the decay — and then pairs it with a confidence-scheduled verifier that casts “how many draft tokens should I actually send to the target model this round?” as a provably lossless, load-aware throughput-maximization problem with an exact greedy solution. Deployed inside DeepSeek-V4’s production serving stack in place of the old single-token MTP-1 drafter, DSpark lifts per-user generation speed by 60-85% (V4-Flash) and 57-78% (V4-Pro) at matched aggregate throughput, and — more importantly — it unlocks strict low-latency service tiers that the old baseline simply could not sustain at all.

Prerequisites: What You Need to Know First

This paper sits squarely in LLM inference systems. I will build up every concept from scratch so that the architecture and scheduling sections that follow are self-contained.

Why Autoregressive Decoding Is Slow

A transformer language model generates text one token at a time. To produce token $x_{n+1}$, it needs a full forward pass over the model conditioned on $x_1, \ldots, x_n$. Once $x_{n+1}$ is sampled, it appends it to the context and repeats. This creates a hard sequential dependency: you cannot compute $x_{n+2}$ before you know $x_{n+1}$.

On modern GPUs this is disastrous for hardware utilization. A single decode step moves the entire model’s weights (hundreds of gigabytes for frontier models) from HBM into the compute units to produce just one new token per sequence. The arithmetic intensity — floating-point operations per byte moved — is tiny, so decoding is memory-bandwidth bound, not compute bound. The GPU’s matrix units, which are what you pay for, sit mostly idle. This is the central inefficiency that speculative decoding exploits: if you can get the model to “commit” to several tokens using only one expensive pass over the weights, you convert idle compute into useful work.

The KV Cache

To avoid recomputing every previous token’s attention keys and values at every step, transformers cache them layer by layer. As context grows, the KV cache grows linearly with context length and consumes GPU memory that would otherwise host a larger batch of concurrent requests. This matters for DSpark because, as we will see in Section 5, the paper’s production deployment explicitly treats “how much decode batch size is available on this GPU right now” as a resource that has to be shared fairly between ordinary requests and speculative-decoding verification.

Multi-Token Prediction (MTP) and Why “MTP-1” Is the Baseline to Beat

DeepSeek’s own models (starting with DeepSeek-V3) are trained with an auxiliary Multi-Token Prediction objective: in addition to predicting the very next token, the model also learns to predict a small number of future tokens using lightweight extra prediction heads attached to the main model. At inference time, these MTP heads can be repurposed directly as a built-in, always-available draft model — no separate drafter needs to be trained or deployed, because the capability is already baked into the target model’s own checkpoint. “MTP-1” specifically means using one MTP head to draft exactly one extra token per round (i.e., $\gamma=1$). This paper’s production baseline is MTP-1 because, historically, deploying a static multi-token drafter with a larger $\gamma$ (e.g., “MTP-3” or “MTP-5”, drafting 3 or 5 tokens unconditionally every round) was found to degrade aggregate throughput under high concurrency — the fixed, larger verification cost was not worth it once many concurrent requests compete for GPU batch capacity. This is exactly the historical failure mode DSpark is designed to avoid: instead of picking one static $\gamma$ that is a compromise between light-load and heavy-load regimes, DSpark can propose a large $\gamma$ and dynamically shrink how much of it gets verified depending on real-time load — getting the upside of a large block during light load without the downside during heavy load that made static MTP-3/5 impractical in production.

Speculative Decoding: Draft, Then Verify

Speculative decoding (Chen et al., 2023; Leviathan et al., 2023) decouples cheap proposal from expensive verification:

1. A lightweight draft model $M_d$ proposes a block of $\gamma$ candidate tokens $x_1, \ldots, x_\gamma$.
2. The expensive target model $M_t$ verifies the entire block in a single forward pass, because attention lets it process all $\gamma$ candidate positions in parallel exactly as if they were already part of the context.
3. Verification uses rejection sampling: at position $k$, accept $x_k$ with probability $\min(1, p_t^k(x_k)/p_d^k(x_k))$, where $p_t^k$ and $p_d^k$ are the target’s and drafter’s distributions at that position. The first rejection at position $k$ discards everything after it, and the target model supplies one freshly-sampled “bonus” token to replace the rejected one.
4. Crucially, this acceptance rule is chosen so that the marginal distribution of the accepted output is mathematically identical to what the target model alone would have produced. Speculative decoding is a pure latency optimization — it is lossless.

Let $\tau$ be the number of tokens accepted per round (including the bonus token), and let $T_{draft}$ and $T_{verify}$ be the wall-clock time of the drafting and verification passes. The average per-token latency is:

$$L = \frac{T_{draft} + T_{verify}}{\tau} \tag{1}$$

This single formula is the organizing principle of the entire paper. There are exactly three ways to make generation faster: shrink $T_{draft}$ (draft cheaper), grow $\tau$ (draft better, i.e., raise the acceptance rate), or shrink the effective $T_{verify}$ (verify only what is worth verifying). Every design decision in DSpark maps onto one of these three levers.

A worked numerical example. Suppose a target model alone takes 20 ms per token (so generating 100 tokens autoregressively takes 2000 ms). Now suppose a drafter proposes $\gamma = 8$ tokens, drafting costs $T_{draft} = 3$ ms, verification of the 8-token block costs $T_{verify} = 25$ ms (verifying a short block in parallel is only slightly more expensive than verifying one token, because the target model’s forward pass is dominated by loading its weights from memory, not by the small amount of extra compute for a few more positions), and on average $\tau = 3.5$ tokens are accepted per round (including the bonus token). Then the per-token latency is $L = (3 + 25)/3.5 \approx 8.0$ ms — roughly a $2.5\times$ speedup over plain autoregressive decoding. Now suppose the drafter is upgraded so that $\tau$ rises to 5.5 while $T_{draft}$ and $T_{verify}$ stay the same: $L = 28/5.5 \approx 5.1$ ms, a $3.9\times$ speedup. This illustrates why the paper is so focused on raising $\tau$ specifically (rather than, say, shrinking $T_{draft}$ further) — once $T_{draft} \ll T_{verify}$, as is the case for parallel drafters, the accepted length $\tau$ is by far the most leveraged quantity in Equation 1.

Continuous Batching and Why Concurrency Changes the Calculus

Production LLM serving systems rarely serve one request at a time. Continuous batching groups many concurrent requests’ decode steps into a single forward pass, so the target model processes a batch of, say, 128 different users’ next-token predictions in one GPU call. This is what makes LLM serving economical: the fixed cost of loading model weights from HBM is amortized across all 128 users’ predictions in that step, rather than paid separately per user.

Speculative decoding interacts with batching in a specific and important way: verifying a block of $\gamma$ draft tokens for one request is, from the target model’s point of view, equivalent to adding $\gamma$ more slots to the batch for that step (one slot per candidate position). If every one of $R$ concurrent requests proposes $\gamma$ tokens, the effective batch size balloons from $R$ to roughly $R \times (1+\gamma)$. Since GPU throughput (measured in steps/second) degrades as batch size grows — first slowly, then more sharply as the batch exceeds the GPU’s compute-bound regime — blindly maximizing $\gamma$ for every request can shrink the batch of requests the GPU can serve per unit time, even though each individual verified block, taken alone, looks efficient. This is precisely the tension DSpark’s hardware-aware scheduler is built to resolve: it treats “how many extra batch slots is it worth spending on this particular draft token” as a first-class scheduling decision rather than a fixed per-request configuration knob.

Autoregressive vs. Parallel Drafters

Autoregressive drafters (small transformer heads that condition each drafted position on the previously sampled draft token, e.g., EAGLE-family models) capture strong sequential dependencies, but their drafting cost scales linearly with block size: $T_{draft} \propto \gamma$. This forces them to stay shallow and use short blocks, or the drafting overhead itself eats the speedup.

Parallel drafters (e.g., Medusa, DFlash) predict all $\gamma$ positions in one forward pass conditioned only on the target model’s context, not on each other. This makes $T_{draft}$ nearly independent of $\gamma$, so parallel drafters can afford deeper networks and much larger blocks under the same latency budget. The catch: because each position marginalizes over all plausible predecessors instead of conditioning on the one token that actually got sampled, parallel drafters are prone to multi-modal collision — e.g., predicting “of” at position 1 and, independently, “problem” at position 2, yielding the incoherent phrase “of problem” when the two locally-plausible continuations were “of course” and “no problem.” This section’s insight is exactly the tension DSpark resolves.

Serving-System Concepts: Throughput, Latency, and SLA

In production, an inference engine batches many concurrent user requests together. Two metrics matter simultaneously: aggregate throughput (total tokens/second produced across all users on a GPU) and per-user generation speed, often reported as tokens-per-second-per-user (TPS). A Service Level Agreement (SLA) specifies a minimum TPS the system must guarantee to each active user. These two metrics trade off: verifying longer speculative blocks can raise a given user’s TPS but consumes GPU batch capacity that could otherwise serve more concurrent users, lowering aggregate throughput. Section 4 of this review shows exactly how DSpark’s scheduler is built to manage this trade-off explicitly rather than ignore it.

Notation Glossary

Because this paper uses a fairly dense set of symbols across the drafting, calibration, and scheduling sections, it helps to have all of them in one place before diving in:

Symbol	Meaning
$M_t$, $M_d$	Target model (large, authoritative) and draft model (small, cheap)
$\gamma$	Number of draft tokens proposed per round (block size)
$\tau$	Number of tokens actually accepted per round (including the bonus token)
$T_{draft}$, $T_{verify}$	Wall-clock time of the drafting pass and the verification pass
$L$	Average latency per generated token, $L = (T_{draft}+T_{verify})/\tau$
$p_t^k$, $p_d^k$	Target and draft probability distributions at draft position $k$
$U_k$	Base logits from the parallel backbone at position $k$
$B_k$	Sequential correction bias injected on top of $U_k$
$h_k$	Backbone hidden state at position $k$
$x_0$	Anchor token (the previous round’s bonus token)
$W_1, W_2$	Low-rank factorization matrices of the Markov head’s bigram bias
$r$	Rank of the low-rank factorization (default 256)
$s_k$	RNN head’s recurrent state at position $k$
$c_k$	Raw confidence head output: predicted probability that position $k$ survives, given the prefix was accepted
$c_k^*$	Analytical (ground-truth) acceptance probability used as the training label for $c_k$
$a_{r,j}$	Cumulative prefix survival probability for request $r$ up to position $j$, $a_{r,j}=\prod_{i\le j} c_{r,i}$
$\ell_r$	Scheduled verification length for request $r$
$R$	Number of concurrent requests in the current batch
$B$ (scheduler context)	Total verification batch size in tokens, $B=\sum_r(1+\ell_r)$
$SPS(B)$	Profiled engine throughput (steps/second) as a function of batch size $B$
$\Theta$	System-wide expected token throughput, $\Theta = \tau \cdot SPS(B)$
$w_k$	Position weight in the training loss, $w_k = \exp(-(k-1)/\gamma)$
$\mathcal{L}_{ce}, \mathcal{L}_{tv}, \mathcal{L}_{conf}$	Cross-entropy, total-variation, and confidence training losses

The Problem: Two Bottlenecks Standing Between Parallel Drafters and Their Theoretical Ceiling

Parallel drafters look like a free lunch: near-zero $T_{draft}$ overhead means you should just make $\gamma$ as large as possible. Two bottlenecks stand in the way.

Bottleneck 1 — Generation quality (suffix decay). Because a parallel drafter predicts every draft position independently and cannot condition on the token actually sampled at earlier positions, its accuracy degrades rapidly deeper into the block. The paper’s own measurements (reproduced as Figure 5 below) show this decay is real and domain-dependent.

Bottleneck 2 — System efficiency (wasted verification). Suppose the drafter proposes 16 tokens. Verifying the whole block costs the target model roughly as much compute as verifying 16 real tokens, regardless of how many of them are actually accepted. If the true expected acceptance is, say, 4 tokens, then verifying 12 more offers little benefit but occupies batch capacity that could have served other concurrent requests. This waste is worst under high concurrency, exactly where serving systems need every drop of GPU throughput.

Figure 1 summarizes the design space and shows where DSpark sits relative to prior drafters.

Figure 1: Drafter Design Space and Where DSpark Sits

graph TD
    A[Speculative Decoding Drafter Design] --> B[Autoregressive Drafters]
    A --> C[Parallel Drafters]
    A --> D[DSpark: Semi-Autoregressive]
    B --> B1["High per-token quality<br/>T_draft grows with gamma<br/>forces small blocks"]
    C --> C1["T_draft nearly independent of gamma<br/>large blocks feasible<br/>suffix decay from independence"]
    D --> D1["Parallel backbone: O_1 draft latency"]
    D --> D2["Lightweight sequential head:<br/>injects intra-block dependency"]
    D --> D3["Confidence-scheduled verifier:<br/>load-aware, lossless truncation"]
    style D fill:#4CAF50,color:#fff
    style D1 fill:#c8e6c9
    style D2 fill:#c8e6c9
    style D3 fill:#c8e6c9

DSpark attacks both bottlenecks with two complementary mechanisms: semi-autoregressive generation (Section 3.1 of the paper, covered next) fixes the quality problem at the algorithm level, and confidence-scheduled verification (Section 3.2, covered further below) fixes the system-efficiency problem at the scheduling level. What makes the paper interesting is that neither mechanism is a heuristic bolted onto the other — the scheduler is derived as an exact solution to a throughput-maximization problem, with a formal proof of losslessness that I will walk through in detail, including the counterexample the authors use to show why a naive version of their own algorithm would silently break correctness.

Architecture Part 1: Semi-Autoregressive Generation

Recap of the DFlash Parallel Backbone

DSpark builds its parallel backbone on top of DFlash (Chen et al., 2026), a state-of-the-art parallel drafter, so it is worth understanding DFlash’s mechanism first. During prefill, DFlash extracts hidden states from a chosen set of target-model layers $\{l_1, \ldots, l_m\}$, concatenates them, and projects them into the draft model’s hidden space:

$$H_{ctx} = \text{RMSNorm}\big(W_c [H^{(l_1)}; \ldots; H^{(l_m)}]\big) \tag{2}$$

These context features are then injected into every draft-model attention layer by concatenating them with the draft block’s own keys and values along the sequence dimension:

$$K_i = [W_i^K H_{ctx};\; W_i^K H_d], \qquad V_i = [W_i^V H_{ctx};\; W_i^V H_d] \tag{3}$$

so every draft position attends bidirectionally to both the rest of the block and to the injected target context. The draft model shares the target model’s (frozen) embedding table and language-modeling head, takes an anchor-token embedding followed by $\gamma$ mask-token embeddings, and emits logits for all mask positions in one forward pass. Because drafting cost is nearly independent of block size, DFlash can afford both deeper networks and larger $\gamma$ than autoregressive drafters at the same latency budget — this is exactly the ”$T_{draft}$ collapses to $O(1)$” property from Equation 1 that makes parallel drafting attractive in the first place.

DSpark makes one small but useful modification to this backbone: instead of feeding an anchor token plus $\gamma$ mask tokens and only predicting the $\gamma$ mask positions, DSpark treats the anchor itself as the first prediction position. So $\gamma$ input tokens (anchor + $\gamma - 1$ masks) yield $\gamma$ draft logits instead of $\gamma + 1$ inputs yielding $\gamma$ logits — a small compute saving with no quality cost, because the anchor’s own probability distribution is already known information that is worth predicting from within the same pass.

Why Parallel Alone Isn’t Enough: The Multi-Modal Collision, Formally

Consider a context that admits two plausible two-token continuations: “of course” and “no problem.” A parallel drafter computes $p_1(\cdot)$ and $p_2(\cdot)$ independently by marginalizing over all predecessors, so the first position might sample “of” (a locally reasonable choice under $p_1$) while the second position, having never seen “of” as an actual sampled token, independently samples “problem” (also locally reasonable under the marginal $p_2$). The resulting draft “of problem” is incoherent — not because either individual token was a bad prediction, but because the joint sequence was never modeled. This failure mode is well known in the non-autoregressive generation literature (Gu et al., 2018) as the price parallel models pay for factorizing the joint distribution into independent marginals instead of a proper autoregressive chain.

The Sequential Correction: An Autoregressive Factorization on Top of Parallel Logits

DSpark’s fix is not to abandon parallel generation but to add a prefix-dependent transition bias $B_k(x_0, x_{<k}, x_k)$ on top of the parallel backbone’s base logits $U_k$. Rather than defining a new, globally normalized energy model (which, as discussed in the Related Work section below, would break the exact-probability requirement of rejection sampling), the sequential stage induces a proper autoregressive factorization directly:

$$P(X \mid x_0) = \prod_{k=1}^{\gamma} p_k(x_k \mid x_0, x_{<k}), \qquad p_k(v \mid x_0, x_{<k}) = \frac{\exp\big(U_k(v) + B_k(x_0, x_{<k}, v)\big)}{\sum_{u \in \mathcal{V}} \exp\big(U_k(u) + B_k(x_0, x_{<k}, u)\big)} \tag{4}$$

Let’s unpack this derivation step by step, because it is the crux of the whole method:

• $U_k(v)$ is the parallel backbone’s base logit for candidate token $v$ at position $k$, computed once, in parallel, for all $k$ and $v$ — this part remains cheap because it is a single forward pass.
• $B_k(x_0, x_{<k}, v)$ is a small correction term computed sequentially, position by position, that biases the logits toward tokens consistent with the actually-sampled prefix $x_{<k}$.
• Adding the bias inside the softmax (rather than, say, re-weighting probabilities after the fact) guarantees that $p_k$ remains a proper normalized distribution over the vocabulary — this matters because speculative decoding’s rejection-sampling rule needs an exact, computable $p_d^k(x)$ for every candidate token, not just a ranking.
• The product $\prod_k p_k(x_k \mid \ldots)$ is the standard autoregressive chain-rule factorization of a joint distribution, but now each factor is “cheap base logit + cheap correction” instead of “full transformer forward pass conditioned on the prefix.”

Because this sampling is now sequential (each $p_k$ depends on the actually-sampled $x_{<k}$), the correction module $B_k$ must be computationally tiny — the paper’s design goal is $T_{sequential} \ll T_{parallel}$, so that even though the sequential stage runs in a loop, the loop is over a lightweight module and the overall draft latency remains dominated by the one-shot parallel pass. This is exactly why DSpark is called “semi-autoregressive”: full autoregression only over a cheap correction signal, layered on top of a parallel expensive backbone.

Instantiation 1: The Markov Head

The simplest correction restricts $B_k$ to a first-order Markov dependency on just the immediately preceding token, $B(x_{k-1}, x_k)$. Naively this is a full $V \times V$ matrix (where $V$ is vocabulary size, typically ~100K+), which would be enormous to store and slow to apply. DSpark approximates it with a low-rank factorization $B = W_1 W_2$ where $W_1 \in \mathbb{R}^{V \times r}$ and $W_2 \in \mathbb{R}^{r \times V}$ with rank $r = 256$ by default:

$$B(x_{k-1}, \cdot) = W_1[x_{k-1}]\, W_2 \in \mathbb{R}^V \tag{5}$$

Here $W_1$ acts as an embedding lookup for the previous token and $W_2$ projects that embedding back into logit space. The design rationale: a full-rank bigram transition table over a 100K+ vocabulary is both wasteful (most token-pairs are irrelevant) and slow (a $V \times V$ lookup per step at generation time), whereas a rank-256 factorization captures the dominant local co-occurrence structure (e.g., “of” boosting “course” and suppressing “problem”) at a fraction of the memory and compute cost, and — crucially — it can be applied as two small matrix multiplications rather than one huge lookup.

Why this specific design, and where it would fail: the obvious alternative is to let the correction depend on more than one previous token (e.g., a full n-gram context), which would capture more structure but reintroduces the $O(\gamma)$ sequential cost that parallel drafting was trying to eliminate in the first place if the correction module becomes heavy. The Markov head’s boundary condition is exactly this: it can only fix local mode collisions (an adjacent pair of tokens that don’t belong together). If the incoherence spans three or more tokens — a phrase whose first and third word are inconsistent but the immediate two-token transitions all look locally fine — the Markov head has no mechanism to catch it. This motivates the second instantiation.

Instantiation 2: The RNN Head

The RNN head relaxes the one-step memory limit by maintaining a recurrent state $s_k$ that accumulates the entire prefix history seen so far within the current draft block. At each step it concatenates the previous state $s_{k-1} \in \mathbb{R}^r$, the previous token’s Markov embedding $W_1[x_{k-1}] \in \mathbb{R}^r$, and the backbone’s hidden state $h_k \in \mathbb{R}^d$ into a single vector $z_k = [s_{k-1}; W_1[x_{k-1}]; h_k] \in \mathbb{R}^{2r+d}$, then applies one gated recurrent update (structurally a minimal GRU-style cell):

$$s_k = \sigma(W_g z_k) \odot s_{k-1} + \big(1 - \sigma(W_g z_k)\big) \odot \tanh(W_c z_k), \qquad B_k(x_{<k}, \cdot) = W_2^\top \tanh(W_o z_k) \tag{6}$$

with $W_g, W_c, W_o \in \mathbb{R}^{r \times (2r+d)}$ all packed into a single linear projection that is split into gate/candidate/output components for efficiency, and $s_0$ initialized to zero. The gate $\sigma(W_g z_k)$ interpolates between “keep the old state” and “write the new candidate content” — this is the standard mechanism GRUs use to let gradients (and, at inference time, information) flow across many steps without vanishing.

Design trade-off, made explicit. The RNN head can in principle capture arbitrarily long-range intra-block dependencies (bounded only by $\gamma$, which is small — typically 5-16), which the Markov head cannot. But the paper’s own ablation (Section 4.3.2, discussed below) finds the RNN head provides only marginal gains over the Markov head, concentrated at longer proposal lengths, while being more complex to implement and deploy. The authors therefore ship the Markov head as the default — a deliberate choice to trade a small amount of theoretical modeling capacity for simplicity and predictable latency in a production system. This is a good illustration of the paper’s overall design philosophy: every component is sized to be “just barely enough” complexity to close the specific gap identified, not maximally expressive.

Architecture Part 2: Confidence-Scheduled Verification

Generating a large, high-quality draft block (Part 1) does not automatically translate into end-to-end speedup, because verifying the entire block is not free. This section is where DSpark becomes as much a systems paper as an algorithms paper.

The Confidence Head

DSpark attaches a lightweight confidence head that predicts, for each draft position $k$, a scalar $c_k \in (0,1)$ interpreted as: the conditional probability that the draft token at position $k$ survives target verification, given that all preceding tokens in the block were accepted. Architecturally this is just a linear projection followed by a sigmoid:

$$c_k = \sigma\Big(w^\top [h_k;\, W_1[x_{k-1}]]\Big) \tag{7}$$

taking as input the backbone’s hidden state $h_k$ and the Markov embedding of the previous draft token. The supervision target is the analytical acceptance rate — not an empirical label, but a closed-form quantity derived directly from speculative-decoding theory. Because the standard acceptance probability at a position equals $1 - \tfrac{1}{2}\lVert p_d - p_t \rVert_1$ (half the total-variation distance between draft and target distributions — this identity comes directly from the rejection-sampling construction in Leviathan et al., 2023), the target label is:

$$c_k^* = 1 - \tfrac{1}{2} \lVert p_k^d - p_k^t \rVert_1 \tag{8}$$

This is an elegant design choice: instead of training the confidence head against noisy empirical accept/reject outcomes (which would require many samples per position to get a reliable label), the authors supervise it directly against the exact quantity that determines acceptance probability under the theory. The obvious alternative — training on binary accept/reject outcomes with a cross-entropy loss — would work but converge more slowly and more noisily, since a single accept/reject sample per training example is a much higher-variance target than the exact analytical rate.

Why Raw Confidence Scores Are Not Enough: The Calibration Problem

A subtlety separates DSpark’s confidence head from prior threshold-based schedulers (e.g., SpecDec++): those only need confidence scores to rank draft tokens correctly (higher score = more likely to survive). DSpark’s hardware-aware scheduler (next subsection) needs the absolute magnitude of the cumulative product $\prod_{i \le k} c_i$ to compute an expected throughput number that can be compared across different candidate verification lengths. But neural network confidence estimates are well known to be systematically overconfident (Guo et al., 2017) — the raw scores discriminate well (the paper reports ROC-AUC of 0.81-0.90) but are miscalibrated (Expected Calibration Error of 3-8%), which would distort the scheduler’s throughput estimate and lead to systematically wrong scheduling decisions.

Sequential Temperature Scaling (STS) is the fix. Because each $c_i$ models a conditional probability, the chain rule says the joint survival probability of a prefix factorizes as the cumulative product $\prod_{i \le k} c_i$. STS calibrates this product position by position, left to right, on a held-out validation set: at each position $k \in \{1, \ldots, \gamma\}$, it runs a simple 1D grid search over a temperature scalar to minimize the Expected Calibration Error of the cumulative product, holding all previously-calibrated positions fixed. Two design details matter here: (1) calibrating the product rather than each $c_k$ in isolation is necessary because errors compound multiplicatively across the chain — a slightly overconfident $c_1$ and $c_2$ together produce a badly overconfident $a_2 = c_1 c_2$; (2) temperature scaling is an order-preserving (monotonic) transformation, so it fixes the magnitude problem without disturbing the ranking the confidence head already learned — this is important because it means calibration is a pure “add-on” that cannot make the underlying discrimination worse.

Deriving the Acceptance-Rate Identity Behind Equation 8

Equation 8 asserts $c_k^* = 1 - \tfrac12\lVert p_k^d - p_k^t\rVert_1$ without proof in the main text, but the identity is short enough to derive completely, and doing so is worth the five lines because it is the load-bearing fact that turns “confidence” into a calibratable, physically meaningful quantity rather than an arbitrary neural-network score.

Start from the rejection-sampling rule itself: draw a candidate $x \sim p_d$, and accept it with probability $\min\!\big(1, p_t(x)/p_d(x)\big)$. The unconditional probability that a freshly drawn draft token is accepted is obtained by marginalizing over the draw:

$$P(\text{accept}) = \sum_{x \in \mathcal{V}} p_d(x)\,\min\!\Big(1, \frac{p_t(x)}{p_d(x)}\Big) = \sum_{x \in \mathcal{V}} \min\big(p_d(x),\, p_t(x)\big) \tag{8a}$$

(the second equality just distributes $p_d(x)$ into the $\min$, using $p_d(x)\cdot 1 = p_d(x)$ and $p_d(x) \cdot p_t(x)/p_d(x) = p_t(x)$). Now use the elementary two-number identity $\min(a,b) = \tfrac12(a + b - |a-b|)$, valid for any real $a, b$, term by term inside the sum:

$$\sum_{x} \min(p_d(x), p_t(x)) = \frac{1}{2}\left(\sum_x p_d(x) + \sum_x p_t(x) - \sum_x |p_d(x) - p_t(x)|\right) \tag{8b}$$

Both $p_d$ and $p_t$ are probability distributions, so $\sum_x p_d(x) = \sum_x p_t(x) = 1$, and by definition the total-variation distance is $\lVert p_d - p_t\rVert_{TV} = \tfrac12\sum_x |p_d(x)-p_t(x)| = \tfrac12\lVert p_d-p_t\rVert_1$. Substituting:

$$P(\text{accept}) = \frac12(1 + 1) - \frac12\lVert p_d - p_t\rVert_1 = 1 - \frac12\lVert p_d - p_t\rVert_1 \tag{8c}$$

which is exactly Equation 8’s target label $c_k^*$. Sanity-check with the paper’s own worked numbers (reused later in the non-anticipation counterexample): $p_t(A)=0.7, p_t(B)=0.3$, $p_d(A)=p_d(B)=0.5$. Direct computation of the left-hand side of (8a): $\min(0.5,0.7) + \min(0.5,0.3) = 0.5 + 0.3 = 0.8$. Via the TV-distance identity (8c): $\lVert p_d-p_t\rVert_1 = |0.5-0.7| + |0.5-0.3| = 0.2+0.2=0.4$, so $1 - \tfrac12(0.4) = 1-0.2=0.8$. The two routes agree, confirming the identity is not just algebraically correct but numerically consistent with the values used elsewhere in this review.

Why this matters beyond being a nice fact. It tells you that “confidence” in DSpark is not a subjective, model-specific notion — it is exactly the quantity that governs how often rejection sampling will keep a token, expressed in a form ($1$ minus a distance between two distributions) that is symmetric, bounded in $[0,1]$, and equal to $1$ exactly when $p_d = p_t$ (perfect drafting) and equal to $0$ exactly when $p_d, p_t$ have disjoint support (worst-case drafting). Training the confidence head to predict this specific closed form, rather than an empirical accept/reject frequency, is what makes the STS calibration step in the next subsection well-posed: you are calibrating a network’s estimate of a known, well-defined statistical distance, not chasing a moving, sampling-noise-dominated target.

STS in Numbers: Per-Position Calibration on the Reliability Diagram

The paper’s reliability diagram (Figure 6, Alpaca dataset) reports per-position ROC-AUC and ECE both before and after STS, at four sampled positions across a block. Reproducing the exact numbers is more informative than the “0.81-0.90 AUC, 3-8% ECE” range quoted earlier, because the pattern across positions is itself diagnostic:

Figure 5b (paper Fig. 6 detail): Per-position discrimination and calibration error.

Position	ROC-AUC	ECE before STS	ECE after STS
1	0.818	5.7%	2.0%
3	0.812	8.2%	1.7%
5	0.864	5.8%	0.8%
7	0.907	3.3%	0.4%

Two patterns are worth calling out explicitly, because neither is obvious a priori. First, discrimination (AUC) improves with position (0.818 → 0.907) even though intuitively later positions should be “harder” to predict, since they depend on a longer, more compounded context. The likely explanation is a selection effect: by construction, position 7 is only ever evaluated on rollouts where positions 1-6 already survived acceptance (recall the position-wise conditional acceptance methodology from the “Why Does Parallel Beat Pure Autoregressive” section) — conditioning on a long accepted prefix selects for an easier, more predictable subpopulation of continuations, so the confidence head’s job at position 7 is implicitly easier given that it is being asked at all. Second, ECE-before-calibration is non-monotonic, peaking at position 3 (8.2%) rather than rising steadily — a reminder that raw miscalibration is not a simple function of sequence depth and that STS’s position-by-position grid search (rather than a single global temperature) is doing real, position-specific work: a single shared temperature tuned to fix position 1 would very likely under- or over-correct positions 3, 5, and 7, given how differently miscalibrated they start out. After STS, ECE is both much smaller and much flatter across positions (2.0%, 1.7%, 0.8%, 0.4%), which is the practical justification for calibrating the cumulative product sequentially rather than applying one blanket correction.

The Hardware-Aware Prefix Scheduler

This is the paper’s most novel contribution, so it is worth deriving in full. The question the scheduler answers is: given a batch of $R$ concurrent requests, each with its own per-position survival probabilities, how many tokens of each request’s draft block should actually be sent to the target model for verification this round, in order to maximize system-wide throughput?

Setup. For request $r$, let $c_{r,1}, \ldots, c_{r,\gamma}$ be per-position confidence estimates, and $\ell_r \in \{0, \ldots, \gamma\}$ the chosen verification length. Because speculative decoding only ever accepts a contiguous prefix, the survival probability of the token at position $j$ is the cumulative product $a_{r,j} = \prod_{i \le j} c_{r,i}$. If the batch verifies $B = \sum_r (1 + \ell_r)$ tokens total (the $+1$ accounts for the anchor/bonus-token slot per request), the target model runs at some measured throughput $SPS(B)$ (steps-per-second, profiled once at engine startup as a lookup table over batch size). The expected accepted tokens in this round is $\tau = \sum_r \big(1 + \sum_{j=1}^{\ell_r} a_{r,j}\big)$, so the system-wide expected token throughput to maximize is:

$$\Theta = \tau \cdot SPS(B) \tag{objective}$$

Why this looks combinatorial, and why it isn’t. Choosing $\ell_1, \ldots, \ell_R$ independently looks like a search over an exponentially large space of length assignments. The key structural insight that collapses this: because $a_{r,j}$ is monotonically non-increasing in $j$ (a longer prefix can only be as likely or less likely to fully survive), the marginal gain in $\tau$ from extending request $r$‘s verification length from $j-1$ to $j$ is exactly $a_{r,j}$ — a single, comparable number, independent of what other requests are doing. This means the incremental value of “verify one more token” is directly comparable across all requests, not just within one request’s own block. So instead of assigning lengths per-request, DSpark pools every candidate token-extension $(r, j)$ across the whole batch into one global list, sorts it by $a_{r,j}$ descending, and greedily admits from the top:

Algorithm 1 — Hardware-Aware Prefix Scheduler (restated with numbered steps):

Input: Active requests r in {1,...,R}; confidence sequences c_{r,1},...,c_{r,gamma};
       profiled throughput curve SPS(B)
Output: Per-request verification lengths l*_1, ..., l*_R

1.  For each request r: compute prefix survival probabilities
        a_{r,j} = product_{i<=j} c_{r,i}    for j = 1,...,gamma
2.  Build candidate pool E = { (r,j) : a_{r,j} > 0 }; sort E descending by a_{r,j}
3.  Initialize l_r = 0 for all r; batch size B = R; expected accepts tau* = R
4.  Initialize best-so-far: Theta_best = R * SPS(R); l*_r = 0 for all r
5.  For each (r,j) in sorted order E:
5a.     l_r = j;  B = B + 1;  tau* = tau* + a_{r,j}
5b.     Theta = tau* * SPS(B)
5c.     If Theta > Theta_best:
            Theta_best = Theta;  l*_r = l_r   for all r   (snapshot current lengths)
        Else:
            break                                          # early stop -- see causality proof below
6.  Return (l*_1, ..., l*_R) achieving Theta_best

Prose walkthrough of the algorithm. Step 1 turns each request’s raw per-position confidences into cumulative survival probabilities — this is the same quantity used for STS calibration. Step 2 is the key move: it flattens the per-request, per-position problem into one global, ranked queue of “verification opportunities,” ordered by how likely each one is to pay off. Steps 3-4 initialize the state as if every request verifies zero extra draft tokens beyond the guaranteed anchor token (this is always a feasible, safe baseline — equivalent to plain one-token decoding). Step 5 is a single linear pass over the globally-sorted candidate list: each iteration tentatively admits the next most-promising extension, recomputes the expected system throughput $\Theta$ under the new (slightly larger) batch size, and either accepts this bigger frontier as the new best (if throughput went up) or stops immediately (if it went down). Step 6 returns whatever configuration achieved the best throughput seen along that single sorted pass.

A worked trace with two requests. To make the algorithm concrete, consider $R=2$ requests with $\gamma=2$ each. Request 1 has confidences $c_{1,1}=0.9, c_{1,2}=0.5$, giving survival probabilities $a_{1,1}=0.9$, $a_{1,2}=0.45$. Request 2 has $c_{2,1}=0.6, c_{2,2}=0.6$, giving $a_{2,1}=0.6$, $a_{2,2}=0.36$. Suppose the profiled capacity curve is $SPS(2)=1.0$, $SPS(3)=0.8$, $SPS(4)=0.55$, $SPS(5)=0.5$, $SPS(6)=0.3$.

1. Initialize: $B=2$ (one anchor slot per request), $\tau^*=2$, $\Theta_{best} = 2 \times SPS(2) = 2.0$.
2. Sorted candidate pool (descending by survival probability): $(1,1){:}0.9$, $(2,1){:}0.6$, $(1,2){:}0.45$, $(2,2){:}0.36$.
3. Admit $(1,1)$: $B=3$, $\tau^*=2+0.9=2.9$, $\Theta = 2.9\times SPS(3) = 2.9\times0.8=2.32 > 2.0$ → accept, new best $\Theta_{best}=2.32$, snapshot $\ell_1=1,\ell_2=0$.
4. Admit $(2,1)$: $B=4$, $\tau^*=2.9+0.6=3.5$, $\Theta=3.5\times SPS(4)=3.5\times0.55=1.925 < 2.32$ → this candidate alone would look bad, but the algorithm does not stop yet in this trace because we must check the next candidate before declaring a break under the illustrative unconstrained-search variant; under the strict causal Algorithm 1 as written, the loop breaks here immediately, returning $\ell_1^*=1, \ell_2^*=0$ with $\Theta_{best}=2.32$.
5. What the extra candidates would have shown (for intuition only, not executed by the causal algorithm): admitting $(1,2)$ next would give $B=5$, $\tau^*=3.5+0.45=3.95$, $\Theta=3.95\times0.5=1.975$ — still below 2.32 — and admitting $(2,2)$ after that would give $B=6$, $\tau^*=3.95+0.36=4.31$, $\Theta=4.31\times0.3=1.293$, even worse. In this particular toy example the early-stop at step 4 happens to also be the global maximum, which is a useful sanity check but is not guaranteed in general on a jagged, non-unimodal $SPS(B)$ — exactly the gap that motivates the production-side adaptation in Section 5.2.

The final decision — request 1 gets 1 extra token verified, request 2 gets none this round — illustrates the core behavior: even though request 2’s first-position confidence (0.6) is individually decent, the marginal system-wide value of extending it is lower than request 1’s, once the shared batch-size cost is accounted for. This is exactly the cross-request comparison that a naive “each request picks its own threshold independently” scheme cannot make.

Why greedy is provably optimal here (not just a heuristic). This is a rare case where a greedy algorithm gives the exact global optimum, and the reason is the monotonicity property noted above: because each request’s own marginal gains $a_{r,1} \ge a_{r,2} \ge \ldots \ge a_{r,\gamma}$ are already sorted in non-increasing order internally, and because the objective’s only dependence between requests is through the shared batch-size term $SPS(B)$, admitting candidates in strict global descending order of marginal value ($a_{r,j}$) is equivalent to, at every possible batch size $B$, having already selected the value-maximizing subset of extensions for that $B$. If the true optimal batch size is $B^*$, the greedy sorted-and-truncated list is by construction the highest-$\tau$ way to reach any given $B$ — so scanning $B$ upward and stopping at the point where $\Theta$ stops increasing recovers the global maximum, provided $\Theta$ is unimodal in $B$ (the paper is explicit that this assumption — a smoothly decaying hardware capacity curve — is what licenses the early-stop rule; Section 5.2, discussed below, revisits what happens when real hardware violates it).

Formal Proof Sketch: Why the Sorted List Reaches the Global Optimum

The intuition above deserves to be made precise, because “greedy is optimal” claims are notoriously easy to state and occasionally wrong in subtly different problem setups. Here is the argument in three steps.

Lemma 1 (prefix-closure is automatic). Let $E = \{(r,j) : r \in \{1,\ldots,R\}, j \in \{1,\ldots,\gamma\}\}$ sorted descending by $a_{r,j}$, and let $G_m$ denote the top-$m$ elements of this sorted list for any $m \in \{0, \ldots, R\gamma\}$. Claim: for every request $r$, if $(r,j) \in G_m$ then $(r, j-1) \in G_m$ too (whenever $j>1$). Proof: because $a_{r,j-1} \ge a_{r,j}$ (survival probability is non-increasing along a prefix — a token can only survive if everything before it also survived), $(r,j-1)$ sorts at or before $(r,j)$ in descending order, so if $(r,j)$ is among the top $m$, $(r,j-1)$ — which ranks at least as high — must be too. This means the greedy list never admits a “hole” (verifying position $j$ without also verifying positions $1,\ldots,j-1$) automatically, without needing to be told to respect the prefix constraint explicitly.

Lemma 2 (top-$m$ by value is the unique value-maximizing choice of that size). Among all subsets $S \subseteq E$ of size $m$, $G_m$ maximizes $\sum_{(r,j)\in S} a_{r,j}$. Proof: standard exchange argument — if some optimal $S^\star \ne G_m$ existed, it must contain an element with value strictly less than some element outside $S^\star$ that is in $G_m$; swapping the two weakly increases the sum, contradicting optimality unless $S^\star = G_m$ already.

Step 3 (assembling the two lemmas). Combining Lemma 1 and Lemma 2: $G_m$ is simultaneously (a) prefix-consistent for every request (so it corresponds to an actual, executable choice of lengths $\ell_1,\ldots,\ell_R$) and (b) the value-maximizing choice of size $m$. So define $\tau^*(B) := R + \sum_{(r,j)\in G_{B-R}} a_{r,j}$ — the true best-achievable expected accepted length at total verification batch size $B$. Because $\Theta(B) = \tau^*(B)\cdot SPS(B)$ by definition, and $\tau^*(B)$ is realized exactly by incrementally growing $G_m$ one element at a time (which is precisely what Algorithm 1’s Step 5 loop does), scanning through the sorted list and evaluating $\Theta$ after each admission traces out the entire curve $\{\Theta(B)\}_{B=R}^{R(1+\gamma)}$ at its true optimum for every $B$. Finding $\max_B \Theta(B)$ over this curve — by scanning all the way through if $\Theta$ is not known to be unimodal, or by stopping at the first decrease if it is — therefore finds the global optimum over all feasible length assignments, not merely a locally good one. The unimodality assumption is used only to justify the early stop; the underlying claim that “the sorted-by-value order gives the truly best $\tau$ at every batch size” (Lemmas 1-2) holds unconditionally.

The Complete $\Theta(B)$ Curve for the Two-Request Worked Example. Applying $\tau^*(B)$ to the $R=2$, $\gamma=2$ example above makes Step 3 concrete — rather than only checking one candidate at a time, here is the entire curve the scan implicitly explores:

$B$	Admitted set $G_{B-2}$	$\tau^*(B)$	$SPS(B)$	$\Theta(B)$
2	$\emptyset$	2.00	1.00	2.00
3	$\{(1,1)\}$	2.90	0.80	2.32
4	$\{(1,1),(2,1)\}$	3.50	0.55	1.925
5	$\{(1,1),(2,1),(1,2)\}$	3.95	0.50	1.975
6	$\{(1,1),(2,1),(1,2),(2,2)\}$	4.31	0.30	1.293

Reading down the $\Theta(B)$ column confirms two things at once: (1) $B=3$ is indeed the unconstrained global maximum over this entire table (not just a local peak relative to its immediate neighbor), so the early-stopping causal scheduler’s answer is exactly correct in this instance; and (2) the curve is unimodal here (rises then falls monotonically), which is exactly the condition Lemma 1-2’s optimality guarantee needs the early-stop shortcut to be safe — Section 5.2 below covers what happens once real GPU throughput curves are not this well-behaved.

The Non-Anticipating Property: A Worked Counterexample

The scheduler must satisfy a subtle correctness constraint that the paper calls the non-anticipating property: the decision to admit draft token $k$ for verification must depend only on information available before token $k$ is sampled, never on the realization of $x_k$ itself. If this is violated, the scheduler silently breaks the losslessness guarantee that makes speculative decoding safe to deploy at all — a serving optimization stops being “free” and starts changing what users actually see.

The paper proves this matters with a concrete numerical counterexample (Appendix A), which I reproduce here because it is the kind of correctness argument that is easy to wave away in prose but very convincing worked out in full.

Take a single request ($R=1$), max block length $\gamma = 2$, first-position survival probability $a_1 = 0.8$, and a profiled capacity curve $SPS(1)=1.0$, $SPS(2)=0.5$, $SPS(3)=0.45$. The candidate throughputs for verifying 0 or 1 tokens are:

$$\Theta_0 = 1 \cdot SPS(1) = 1.0, \qquad \Theta_1 = (1+0.8)\cdot SPS(2) = 0.9 \tag{9}$$

So far, $\Theta_1 < \Theta_0$ — a correct causal scheduler with early-stopping halts right here and commits to $\ell = 0$, before ever looking further ahead. But suppose instead the scheduler is allowed to look one step further (i.e., no early stop) and evaluates $\Theta_2$ too. Because the Markov confidence head’s next score $c_2$ depends on the token $x_1$ that was actually sampled, two realizations of $x_1$ give two different values of $a_2 = a_1 c_2$:

• If $x_1$ happens to yield $c_2 = 0.9$: $a_2 = 0.8 \times 0.9 = 0.72$, so $\Theta_2 = (1+0.8+0.72)\times 0.45 = 1.134$ — now the global maximum, so the un-early-stopped scheduler would retroactively decide $\ell=2$, admitting $x_1$.
• If $x_1$ happens to yield $c_2 = 0$: $a_2 = 0$, so $\Theta_2 = (1+0.8+0)\times 0.45=0.81 < \Theta_0$, so the scheduler decides $\ell=0$, rejecting $x_1$.

The punchline: whether $x_1$ is even offered for verification now depends on the value of $x_1$ itself — a textbook violation of non-anticipation. The paper turns this into a distributional argument: with $p_t(A)=0.7, p_t(B)=0.3$ and $p_d(A)=p_d(B)=0.5$ (so the honest acceptance probability at position 1 is $\min(0.7,0.5)+\min(0.3,0.5)=0.8$, matching the assumed $a_1$), suppose (as in the example) $x_1=A$ triggers the high-confidence path ($\ell=2$, admitted) while $x_1=B$ triggers the low-confidence path ($\ell=0$, rejected and resampled from $p_t$). Then:

$$\Pr(Y=A) = \Pr(x_1=A)\cdot 1 + \Pr(x_1=B)\cdot p_t(A) = 0.5 + 0.5\times 0.7 = 0.85 \tag{10}$$

which is not equal to the target’s true $p_t(A) = 0.7$ — the retrospective scheduler has silently corrupted the output distribution. This is exactly why Algorithm 1’s early-stopping break is not a performance shortcut but a correctness requirement: by halting the search the instant $\Theta$ stops improving, the admission decision for position $k$ never touches information ($c_{k+1}$ or later) that depends on the realization of $x_k$.

Figure 2 visualizes the scheduling pipeline end-to-end, including where calibration and the causality constraint sit.

Figure 2: DSpark Decode Cycle — From Draft to Scheduled Verification

sequenceDiagram
    participant T as Target Model
    participant P as Parallel Backbone
    participant S as Sequential Head
    participant C as Confidence Head + STS
    participant Sch as Hardware-Aware Scheduler

    T->>T: Generate anchor token D (previous round's bonus)
    T->>P: Anchor D as input
    P->>P: One forward pass -> base logits U_1..U_gamma
    P->>S: Hidden states h_1..h_gamma
    loop k = 1 to gamma (lightweight, sequential)
        S->>S: Sample x_k from p_k(base logit + bias B_k)
        S->>C: h_k, previous token embedding
        C->>C: Compute raw c_k, apply STS calibration
    end
    C->>Sch: Calibrated confidences c_1..c_gamma for this + other requests
    Sch->>Sch: Pool all (r,j) candidates, sort by a_{r,j}, greedy admit with early-stop
    Sch->>T: Scheduled prefix length l*_r per request
    T->>T: Verify only the scheduled prefix (single parallel pass)
    T->>T: Accept longest valid prefix, emit bonus token for next round

A Complexity Analysis: Quantifying the “Semi” in Semi-Autoregressive

The ablation results reported later (Section “Latency overhead”) measure that scaling the sequential loop from 4 to 16 draft positions adds only 0.2-1.3% to full-round latency. That number is easy to accept on faith but more convincing when it falls out of an explicit FLOP count, so it is worth deriving why the sequential correction is cheap by construction, not just cheap in this one measurement.

Cost of one Markov-head step. Recall Equation 5: $B(x_{k-1}, \cdot) = W_1[x_{k-1}]\,W_2$, with $W_1 \in \mathbb{R}^{V\times r}$, $W_2 \in \mathbb{R}^{r \times V}$, and default rank $r=256$. Looking up a row of $W_1$ is a free memory read (no FLOPs); the subsequent multiplication of that length-$r$ row against $W_2$ is a single vector-matrix product of size $r \times V$, costing $2rV$ floating-point operations (the factor of 2 counts one multiply and one add per entry). Over a full block of $\gamma$ positions, run sequentially, the total sequential-head cost is:

$$\text{FLOPs}_{seq} = 2\,\gamma\, r\, V \tag{15}$$

Plugging in production-scale numbers. With $r=256$ (default), a representative frontier-model vocabulary $V \approx 1.28\times10^5$, and the production block size $\gamma=5$: $\text{FLOPs}_{seq} = 2 \times 5 \times 256 \times 128{,}000 \approx 3.3\times10^8$ — about 0.33 GFLOPs per request per round. This is a genuinely small number in modern LLM-serving terms.

Cost of the parallel backbone, for comparison. A standard order-of-magnitude estimate for a transformer(-like) forward pass is roughly $2 \times N_{active} \times n_{tok}$ FLOPs, where $N_{active}$ is the number of active parameters a token touches (for an MoE layer, this is the sparse, routed subset, not the full parameter count) and $n_{tok}$ is the number of tokens processed in that pass. The paper does not disclose the exact parameter count of the three co-deployed MoE backbone layers, but even a deliberately conservative illustrative estimate — treating the backbone as touching on the order of a few hundred million to low billions of active parameters per token, processing a block of $\gamma=5$ positions per request — puts the backbone’s forward-pass cost at tens of GFLOPs or more per request per round: three to four orders of magnitude larger than the $\sim 0.33$ GFLOP sequential-head cost computed above.

Why this gap, and not just “it’s small,” is the right way to think about it. The gap arises from where each computation spends its FLOPs: the sequential head’s cost scales with $r$ (a deliberately small rank, $256 \ll d$ for any modern hidden dimension $d$), while the backbone’s cost scales with the full hidden width and depth of a transformer/MoE block. Because $r \ll d$ by roughly one to two orders of magnitude, and because the backbone additionally pays a depth multiplier the sequential head does not, the total gap compounds multiplicatively rather than additively. This is precisely why the measured empirical overhead (0.2-1.3%, from a real profiled system) lands in that range rather than, say, 10-20%: the architecture was designed so that the “autoregressive” half of “semi-autoregressive” only ever touches a rank-256 bottleneck, never the full model width.

Where this estimate would break down. The analysis also tells you exactly where the design would stop being cheap. If a deployment used the full-rank alternative explicitly rejected in the “Instantiation 1: The Markov Head” discussion above ($r=V\approx1.28\times10^5$ instead of $r=256$), Equation 15 would blow up to $2\gamma V^2 \approx 2\times5\times(1.28\times10^5)^2 \approx 1.6\times10^{11}$ FLOPs — comparable to or exceeding the backbone’s own cost, eliminating the entire “cheap correction on top of an expensive backbone” premise the paper is built on. Similarly, quadrupling $\gamma$ to 20 while holding $r$ fixed only grows the sequential cost linearly (Equation 15 is linear in $\gamma$), so large block sizes remain safe — the real danger, as this derivation makes explicit, is in the rank $r$, not the block length $\gamma$, which is exactly why the ablation in Section 4.3.2 varies $\gamma$ freely but the paper never explores $r$ much larger than 256.

Training Objective

DSpark trains on randomly-sampled $\gamma$-token anchor blocks drawn from target-model-generated sequences, with the target model kept entirely frozen and the draft model sharing (and freezing) the target’s embedding table and LM head — only the backbone drafter, sequential head, and confidence head are updated. The overall loss combines three terms, each position-weighted by $w_k = \exp(-(k-1)/\gamma)$ (following Chen et al., 2026), which front-loads training signal onto earlier block positions — a deliberate choice because, under prefix-based verification, an error at position 1 invalidates everything after it, so a training signal that is uniform across positions would over-invest in correctly predicting tokens that, in expectation, rarely get reached anyway.

Why an exponential decay weight, specifically, and not something simpler? The obvious alternative is a linear decay, $w_k = 1 - (k-1)/\gamma$, which also front-loads weight onto position 1 and is arguably easier to reason about. The exponential form $w_k=\exp(-(k-1)/\gamma)$ differs in one important way: it decays multiplicatively, so the weight ratio between any two adjacent positions, $w_{k+1}/w_k = \exp(-1/\gamma)$, is the same constant regardless of which position $k$ you are at, whereas a linear schedule’s adjacent-position ratio changes throughout the block (it shrinks fastest, proportionally, near the end). This constant-ratio property mirrors the actual statistics of prefix-based acceptance: recall that the probability of even reaching position $k$ is itself a product of the previous positions’ survival probabilities, i.e., roughly geometric in $k$ under a stationary per-position acceptance rate — an exponential training weight is the natural match to a quantity ($a_{r,k}$ from the scheduler section) that is itself a running product, whereas a linear weight has no such correspondence to the underlying process. The boundary case where this reasoning breaks down: if per-position acceptance rates were not roughly stationary (e.g., a domain where the last position of a block is systematically much easier than the middle, as could happen with structured, fixed-length output formats), a fixed exponential-decay schedule would still under-weight that easy-but-late position relative to what its actual reachability probability would justify — the paper does not explore adapting $w_k$‘s decay rate per-domain, treating $\gamma$ itself (which appears in the exponent’s denominator) as the only knob that changes the schedule’s steepness.

Cross-entropy loss — standard next-token prediction against the ground-truth sampled token $x_k^*$:

$$\mathcal{L}_{ce} = -\sum_{k=1}^{\gamma} w_k \log p_k^d(x_k^*) \tag{11}$$

Distribution-matching (total-variation) loss — directly penalizes the gap between draft and target distributions:

$$\mathcal{L}_{tv} = \sum_{k=1}^{\gamma} w_k \lVert p_k^d - p_k^t \rVert_1 \tag{12}$$

The intuition for including this in addition to cross-entropy: cross-entropy only pushes probability mass toward the single sampled token, but the quantity that actually determines the speculative-decoding acceptance rate is the full total-variation distance between the two distributions (recall Equation 8: acceptance probability $= 1-\tfrac12\lVert p^d-p^t\rVert_1$). Minimizing $\mathcal{L}_{tv}$ directly optimizes the thing the paper actually cares about, rather than optimizing a proxy (next-token accuracy) that only loosely correlates with acceptance rate.

Confidence loss — binary cross-entropy training the confidence head toward the analytical soft label $c_k^*$ from Equation 8:

$$\mathcal{L}_{conf} = -\sum_{k=1}^{\gamma} w_k \Big[c_k^* \log c_k + (1-c_k^*)\log(1-c_k)\Big] \tag{13}$$

Combined objective, with default weights $\alpha_{ce}=0.1$, $\alpha_{tv}=0.9$, $\alpha_{conf}=1.0$ (note $\mathcal{L}_{tv}$ dominates $\mathcal{L}_{ce}$ by design, consistent with the reasoning above that TV distance is the more directly relevant quantity):

$$\mathcal{L} = \alpha_{ce}\mathcal{L}_{ce} + \alpha_{tv}\mathcal{L}_{tv} + \alpha_{conf}\mathcal{L}_{conf} \tag{14}$$

Experiments: Offline Benchmarks

Setup

Target models: Qwen3-4B/8B/14B and Gemma4-12B. Baselines: Eagle3 (autoregressive, training-time-test) and DFlash (parallel). For fairness, all drafters are retrained in the same framework on the same data — Open-PerfectBlend (1.3M instruction samples: 39.4% math, 38.9% code, 17.6% chat, 4.1% instruction-following), with responses regenerated by each target model itself (so the drafter learns the target’s actual output distribution, not some other model’s). Evaluation spans three domains: math (GSM8K, MATH500, AIME25), code (MBPP, HumanEval, LiveCodeBench), and chat (MT-Bench, Alpaca, Arena-Hard), measuring accepted length $\tau$ per decoding round with the confidence scheduler disabled, so this section isolates pure draft quality from scheduling effects.

Main Result

Table 1 (reproduced from paper Table 1): Accepted length $\tau$ per decoding round.

Target	Drafter	GSM8K	MATH500	AIME25	MBPP	HumanEval	LCB	MT-Bench	Alpaca	Arena-Hard
Qwen3-4B	Eagle3	5.14	4.62	3.92	3.69	4.16	3.77	2.39	2.26	2.55
Qwen3-4B	DFlash	5.40	4.85	4.15	4.40	4.74	4.18	3.07	2.96	2.83
Qwen3-4B	DSpark	6.11	5.70	4.89	5.13	5.38	4.86	3.64	3.54	3.29
Qwen3-8B	Eagle3	5.30	4.77	3.91	3.96	4.33	4.17	2.66	2.54	2.54
Qwen3-8B	DFlash	5.33	4.91	4.07	4.36	4.64	4.39	3.11	2.98	2.81
Qwen3-8B	DSpark	6.17	5.78	5.01	5.16	5.52	5.17	3.72	3.58	3.21
Qwen3-14B	Eagle3	5.24	4.60	3.71	3.81	4.14	4.01	2.62	2.47	2.48
Qwen3-14B	DFlash	5.41	4.84	3.98	4.44	4.59	4.33	3.10	2.94	2.72
Qwen3-14B	DSpark	6.21	5.74	4.94	5.26	5.43	5.02	3.70	3.58	3.13
Gemma4-12B	Eagle3	5.87	5.46	4.83	4.72	5.37	4.16	3.19	3.06	2.72
Gemma4-12B	DFlash	5.45	5.04	4.22	4.39	4.95	3.70	2.98	2.84	2.59
Gemma4-12B	DSpark	6.05	5.78	5.12	5.11	5.64	4.51	3.49	3.35	2.92

DSpark improves macro-average accepted length over Eagle3 by 30.9%/26.7%/30.0% (4B/8B/14B) and over DFlash by 16.3%/18.4%/18.3%, and the gains transfer to a different model family (Gemma4-12B), suggesting the mechanism is architecture-agnostic rather than tuned to one target model’s quirks. Note also the domain effect visible in every row: math and code consistently produce higher accepted length than chat (e.g., Qwen3-4B: 5.57 avg on math vs. 3.49 on chat) — structured tasks are simply more predictable, which is precisely the observation that motivates not using a fixed verification length across domains.

Why Does Parallel (and Semi-Autoregressive) Beat Pure Autoregressive Here? A Position-wise Analysis

This result is counter-intuitive on its face — shouldn’t a model that explicitly conditions on previously-sampled tokens (Eagle3) always beat one that doesn’t (DFlash)? The paper answers this with a position-wise conditional acceptance analysis: for position $k$, measure acceptance rate only among the subset of rollouts where positions $1, \ldots, k-1$ were already accepted — this isolates position $k$‘s intrinsic predictive quality from the compounding effect of earlier rejections.

Figure 3 (paper Fig. 2): Position-wise conditional acceptance by domain.

graph LR
    subgraph Math["Math domain"]
    M1["Pos 1: DFlash 0.88, Eagle3 0.81"] --> M2["Pos 7: DFlash decays, Eagle3 stable/rises"]
    end
    subgraph Chat["Chat domain"]
    C1["Pos 1: DFlash 0.72, Eagle3 0.53"] --> C2["Pos 7: DFlash drops to ~0.63-0.72,<br/>Eagle3 rises to ~0.74"]
    end
    style M1 fill:#4CAF50,color:#fff
    style C1 fill:#4CAF50,color:#fff

Two effects explain the paradox:

1. The Capacity Advantage at Position 1. At the very first draft position, both architectures predict purely from the target context, with no intra-block dependency yet in play — the difference is pure model capacity. Because parallel drafters pay $O(1)$ drafting cost, they can afford deeper networks than autoregressive drafters (which pay $O(\gamma)$ and must stay shallow), so DFlash starts noticeably ahead: 0.88 vs. 0.81 on Math, 0.72 vs. 0.53 on Chat. Since speculative decoding is a strict prefix-matching process, an error at position 1 wipes out the entire block’s potential — so this first-token advantage disproportionately determines the global accepted length, even though DFlash loses ground later.
2. The Limitation of Independence at Later Positions. Deeper into the block, Eagle3’s explicit conditioning lets it exploit the fact that once early tokens lock in a semantic path, later tokens become more predictable — its conditional acceptance holds steady or even rises (0.53 → 0.74 on Chat). DFlash’s independent marginalization cannot exploit this and decays instead (0.87 → 0.78 on Code; 0.72 → 0.63 on Chat) — this is the multi-modal collision failure mode made empirically visible.

DSpark is explicitly engineered to get both effects at once: it inherits DFlash’s high position-1 capacity (starting at 0.93 on Math) while the lightweight sequential head suppresses the later-position decay, giving a curve that starts high and stays high — the best of both regimes rather than a compromise between them.

DSpark vs. Prior Drafters: A Direct Comparison

It is useful to line up DSpark against the two families of prior-art drafters along the axes that actually determine end-to-end speedup, rather than treating “accepted length” as the only number that matters.

Figure 3b: Comparative summary of drafter families.

Property	Autoregressive (Eagle3)	Parallel (DFlash)	Semi-Autoregressive (DSpark)
$T_{draft}$ scaling with $\gamma$	$O(\gamma)$ — linear	$O(1)$ — constant	$O(1)$ backbone + tiny $O(\gamma)$ correction loop
Position-1 accepted quality	Lower (shallow net, e.g. 0.81 on Math)	Higher (deep net, e.g. 0.88 on Math)	Higher (inherits parallel backbone, e.g. 0.93 on Math)
Suffix (late-position) behavior	Stable or improving	Decays (multi-modal collision)	Decay largely suppressed by sequential head
Max practical block size	Small (latency-limited)	Large	Large
Verification policy	Usually fixed / tree-based	Usually fixed-length	Confidence-scheduled, load-aware
Correctness under scheduling	N/A (typically no dynamic scheduler)	N/A	Provably non-anticipating (Appendix A)
Macro accepted-length vs. Eagle3 (Qwen3-4B)	baseline	+6.6% to +8.3% (computed from Table 1 row deltas)	+30.9%
Macro accepted-length vs. DFlash (Qwen3-4B)	—	baseline	+16.3%

Two things stand out from this table that are easy to miss when reading the paper’s prose alone. First, DFlash’s own improvement over Eagle3 (roughly 6-8% macro accepted length on Qwen3-4B, derivable from Table 1) is much smaller than DSpark’s improvement over Eagle3 (30.9%) — meaning the sequential correction head contributes more to the total gain than the switch from autoregressive to parallel drafting did in the first place. Second, none of the prior drafters in this comparison include a scheduling mechanism at all; DSpark’s advantage in the online production setting (Section 5) compounds the offline accepted-length gain with an entirely separate, orthogonal source of improvement (verification-length scheduling), which is why the end-to-end production speedup (60-85%) is larger than what the offline accepted-length numbers alone would predict via Equation 1.

Ablation: A Little Autoregression Goes a Long Way

Drafter depth. Fixing block size at 7 and varying DSpark’s transformer depth from 1 to 5 layers (compared against a 5-layer DFlash baseline), accepted length improves monotonically with depth, with the steepest marginal gain from 1→2 layers. Notably, a 2-layer DSpark already outperforms the 5-layer DFlash baseline across all three domains — i.e., a small amount of sequential correction is worth more than 3 extra layers of pure parallel capacity. This is a meaningful result for anyone deploying under tight parameter or latency budgets: it says the marginal return on “more parallel depth” is lower than the marginal return on “a little bit of sequential dependency modeling.”

Proposal length. Fixing depth at 5 layers and scaling block size $\gamma+1 \in \{4,8,12,16\}$, DSpark’s advantage over DFlash widens as $\gamma$ grows: 16%/15%/18% (math/code/chat) at $\gamma=7$, expanding to 30%/26%/22% at $\gamma=15$. This makes sense given the position-wise analysis above — DFlash’s marginal utility per additional draft token shrinks as suffix decay compounds, while DSpark’s sequential correction keeps paying off further into the block. The RNN head gives only marginal additional gains over the Markov head, mostly at the longest proposal lengths — consistent with the earlier discussion that the RNN head’s extra long-range memory capacity is a “nice to have,” not the dominant factor, which is why the Markov head remains the shipped default.

Latency overhead. Measured at batch size 128 across context lengths {512, 1024, 2048, 4096}, scaling draft length from 4 to 16 adds only 0.2-1.3% to full-round latency relative to DFlash, because the target model’s verification pass dominates total compute at this batch size — the sequential loop’s cost is genuinely negligible in absolute terms, even though it delivers up to a 30% accepted-length improvement.

Figure 4 (paper Fig. 4): Accepted length vs. proposal length, and latency overhead.

graph TD
    A["Proposal length gamma = 4"] --> A1["DSpark +16pct over DFlash (math)"]
    B["Proposal length gamma = 15"] --> B1["DSpark +30pct over DFlash (math)"]
    C["Latency overhead vs DFlash"] --> C1["+0.2pct at gamma=4"]
    C --> C2["+1.3pct at gamma=16"]
    style B1 fill:#4CAF50,color:#fff
    style C1 fill:#c8e6c9
    style C2 fill:#c8e6c9

Verify Smarter, Not Longer: Validating the Confidence Head in Isolation

Before deploying the full hardware-aware scheduler, the authors validate the confidence head alone with a static-threshold sweep (Figure 5, reproduced below): as the confidence threshold rises, acceptance rate among verified tokens rises correspondingly, because the estimator is correctly filtering out tokens that would ultimately be rejected. The effect is domain-dependent, exactly as the domain-variance argument would predict: on Chat, raising the threshold lifts the realized acceptance rate from 45.7% to 95.7% (a huge amount of pruning is available because chat has many low-confidence suffix tokens), while Math and Code — already high-acceptance domains — see milder gains (76.9%→92.5% and 67.6%→92.0% respectively, since there is less “waste” to prune in the first place).

Figure 5 (paper Fig. 5): Confidence threshold sweep — acceptance rate vs. pruning.

Domain	Threshold 0 (no pruning)	Threshold high (aggressive pruning)
Math	76.9% acceptance	92.5% acceptance
Code	67.6% acceptance	92.0% acceptance
Chat	45.7% acceptance	95.7% acceptance

The calibration reliability diagram (paper Figure 6) confirms the motivation for STS: raw confidence has strong discrimination (ROC-AUC 0.81-0.90) but is overconfident (ECE 3-8%); after STS, average ECE drops to ~1%, restoring trustworthy absolute probability estimates for the scheduler’s throughput arithmetic.

Real-World Deployment in DeepSeek-V4

This is where the paper moves from “a good idea” to “a validated production system,” and it is worth reading closely because the engineering adaptations reveal real gaps between the clean theoretical Algorithm 1 and what actually runs on GPUs at scale.

Scalable Training Infrastructure

The production draft models are co-deployed with DeepSeek-V4-Flash and V4-Pro (preview). The parallel backbone uses three MoE layers with mHC and sliding-window attention of 128, block size $\gamma=5$, and the Markov head. Two systems-level training optimizations matter:

• Hidden-state communication instead of full-vocabulary logit communication. Naively, training the drafter requires the target model’s output distribution at every position — but transmitting full-vocabulary logits ($V \approx 10^5$) across parallel workers is a serious bandwidth bottleneck. Instead, DSpark caches the target model’s pre-LM-head hidden states and only projects to logits locally on the draft-model workers, for the sampled positions only. This drops per-token communication complexity from $O(V)$ to $O(d)$ (hidden dimension) — a design that trades a small amount of redundant local compute (recomputing the LM-head projection) for a large reduction in network traffic, which is exactly the right trade when compute is cheap relative to inter-node bandwidth in a large training cluster.
• Anchor-bounded sequence packing. Rather than padding variable-length training sequences (which wastes compute on padding tokens), DSpark samples a fixed number of anchor positions per sequence and packs the resulting independent $\gamma$-token blocks densely using token-level attention indices instead of 2D masks — preserving exact causal masking across many packed, unrelated blocks in one batch without the memory and compute overhead of padding.

Adapting Algorithm 1 to Real Hardware: Two Conflicts

Deploying the clean Algorithm 1 directly exposes two mismatches with production infrastructure:

Conflict 1 — Jagged, non-smooth hardware capacity curves. Algorithm 1’s optimality proof assumed a smoothly decaying, unimodal $SPS(B)$. Real GPU throughput curves are discrete and step-wise (performance cliffs at specific batch sizes due to kernel tiling, memory-bank conflicts, etc.), which can trap the naive greedy early-stop in a local minimum that isn’t the true optimum just beyond the next “cliff.”

Conflict 2 — Incompatibility with continuous CUDA graph replay. Modern high-throughput serving relies on CUDA graph replay and Zero-Overhead Scheduling (ZOS), both of which require the next step’s batch size to be known before the current step finishes executing. But Algorithm 1 as written computes the schedule using the current step’s freshly-sampled confidence scores — a synchronous dependency that would stall the GPU pipeline waiting for the schedule to be computed.

The Asynchronous Fix, and Why It Stays Lossless

DSpark resolves both conflicts with one adaptation: it approximates the upcoming verification capacity limit $K$ using confidence-head outputs from two steps prior, while still sorting the current step’s actual candidate tokens by their up-to-date, current cumulative confidence scores. Only the truncation length $K$ (the batch-capacity ceiling) is decided from stale (two-steps-old) information; which specific tokens fill that budget is still decided by current, accurate confidence ranking. This is a clean separation of concerns: how much budget is available is a hardware-scheduling question that can tolerate a short prediction lag, while who gets the budget is a correctness-sensitive ranking question that must use live data.

Why does this restore causality safety even though it removes the early-stopping break (necessary to avoid getting stuck in local minima from Conflict 1, allowing an unconstrained global search over the jagged curve)? Because the truncation length $K$ now depends only on information available two steps prior, it structurally cannot depend on the realization of the current step’s draft token $x_{r,k}$ — the exact quantity that caused the non-anticipation violation in the counterexample above. In other words, the two-step lag is the causal barrier, replacing the synchronous early-stop as the mechanism that keeps the scheduler lossless, while also making the schedule computable ahead of time for CUDA graph replay.

Physical Execution: Variable-Length Batches Without Padding Waste

A dynamically-scheduled batch produces variable per-request verified-prefix lengths, which clashes with decode kernels heavily optimized for fixed query length. DSpark’s fix is to flatten all tokens across all requests into one physically homogeneous stream (every token treated identically at the kernel level), and encode the logical intra-sequence structure (which tokens belong to which request, and their causal order) via a separate marker tensor consumed by the sparse-attention kernel. On the DeepSeek-V4 architecture specifically, only the index-attention and compression kernels needed modification — a relatively contained engineering footprint for enabling fully dynamic, per-request verification lengths.

Figure 6: End-to-End Production Pipeline.

graph TD
    A[Live traffic: R concurrent requests] --> B[Parallel backbone + sequential head<br/>produce draft tokens + confidences]
    B --> C[Confidence from 2 steps prior<br/>determines batch capacity K]
    C --> D[Current-step confidences sorted,<br/>top-K admitted per request]
    D --> E[Flattened variable-length batch<br/>marker tensor encodes structure]
    E --> F[Target model verification<br/>index-attention + compress kernels]
    F --> G[Accept longest valid prefix<br/>emit bonus token, next round]
    G --> A
    style C fill:#ffe0b2
    style D fill:#c8e6c9

Production Results

Deployed against the prior single-token MTP-1 baseline (which was the production default precisely because static multi-token drafters like MTP-3/5 were found to degrade throughput under high concurrency due to excessive verification overhead — a real prior failure mode that motivates why a dynamic scheduler, not just a bigger static block, was necessary):

Figure 7 (paper Fig. 7): Aggregate throughput vs. per-user TPS — the Pareto frontier.

Model	SLA anchor	Aggregate throughput gain	Notes
V4-Flash	80 tok/s/user (moderate)	+51%	stable comparison regime
V4-Flash	120 tok/s/user (strict)	nominally +661%	MTP-1 nearly collapses here; interpret as “frontier extension,” not literal multiplier
V4-Pro	35 tok/s/user (moderate)	+52%	stable comparison regime
V4-Pro	50 tok/s/user (strict)	nominally +406%	same caveat as above
Both	matched throughput levels	+60-85% (Flash) / +57-78% (Pro) per-user speed	the headline, most representative number

The authors are explicit that the large “nominal” percentages at strict SLAs should be read as “DSpark makes an interactivity tier feasible where the baseline was barely functional,” not as a literal 4-7x multiplicative speedup under normal operating conditions — a rare and welcome bit of self-restraint in how a systems paper reports its most eye-catching numbers.

Figure 8 (paper Fig. 8): Load-adaptive scheduling behavior. Under moderate concurrency (below ~200 concurrent requests for V4-Flash, ~150 for V4-Pro), the scheduler extends verification budgets from MTP-1’s static 2 tokens to roughly 4-6 tokens per request, directly producing the throughput gains above. As concurrency rises toward saturation, the scheduler smoothly contracts this budget, pruning low-confidence draft tokens before they can consume batch capacity needed by other requests — this is the “load-aware” half of the design paying off exactly as intended, visible directly in production telemetry rather than only in simulation.

graph TD
    A["Low concurrency<br/>e.g. under 50 requests"] --> A1["Verification budget: ~5-6 tokens/request<br/>Spare GPU compute -> spend it on speculation"]
    B["Moderate concurrency<br/>e.g. 100-150 requests"] --> B1["Verification budget: ~4 tokens/request<br/>Still net-positive to verify long prefixes"]
    C["High concurrency<br/>e.g. 200+ requests, near saturation"] --> C1["Verification budget: shrinks toward MTP-1's 2 tokens<br/>Protect batch capacity for more requests"]
    A1 --> D["Scheduler output: smooth, monotone<br/>budget-vs-load curve, no manual tuning"]
    B1 --> D
    C1 --> D
    style A1 fill:#c8e6c9
    style C1 fill:#ffe0b2
    style D fill:#4CAF50,color:#fff

Stated Limitations

The paper is upfront about one limitation: the prefix scheduler minimizes wasted verification, but it cannot recover the fixed, unrecoverable draft-side compute spent generating the initial $\gamma$-token block via the parallel backbone. For queries with inherently low acceptance rates (e.g., very open-ended or unusual chat turns), this upfront drafting cost is paid regardless of how aggressively the scheduler later prunes verification. The authors suggest difficulty-aware early exiting within the draft model itself as future work to let such requests skip full-block generation.

Related Work: Where DSpark Sits

Speculative decoding algorithms. The field has moved from early blockwise methods and standalone small-model drafters (Chen et al., 2023; Leviathan et al., 2023) toward multi-token heads integrated into the target model (Medusa, EAGLE family, MTP). A separate thread pursues parallel/blockwise generation to remove the sequential drafting bottleneck entirely (Medusa, P-EAGLE, PARD, DART, DFlash) — this is the lineage DSpark’s backbone descends from.

System-aware scheduling. A parallel thread of work adapts draft/verification length dynamically using confidence heuristics or learned acceptance predictors (SpecDec++, and others), or frames the problem as goodput/latency optimization under real-time load. DSpark’s distinguishing contribution here is the provably lossless, globally-optimal greedy formulation with an explicit non-anticipation proof — most prior confidence-based schedulers are heuristic threshold rules without a corresponding correctness argument.

Parallel/non-autoregressive generation. The tension between parallel generation speed and joint-sequence coherence traces back to Non-Autoregressive Transformers (Gu et al., 2018), which pioneered predicting all positions independently and immediately ran into the same mode-averaging problem DSpark addresses. Two broad remedies have been explored in the wider NAT literature: steering all positions toward a single consistent output via latent variables, or reintroducing limited sequential structure (iterative refinement, block-level autoregression, or structured output layers like CRF/CTC/HMM). DSpark’s contribution the paper is careful to distinguish: speculative decoding additionally requires exact, per-token probabilities for rejection sampling, which rules out most of these NAT remedies outright — global partition-function models (CRF-NAT) cannot provide exact per-token probabilities, and latent-marginalization-based models (CTC-drafter) are restricted to greedy verification. DSpark’s local, factorized correction (Equation 4) is specifically designed to remain an exact softmax at every position, which is what makes it compatible with lossless rejection sampling in the first place — a nice example of a design constraint (needing exact probabilities) ruling out an entire family of otherwise-reasonable architectural choices.

To make the positioning concrete, here is how DSpark’s individual design choices map onto the broader literature it draws from or distinguishes itself from:

• Drafter backbone lineage: DFlash (Chen et al., 2026) → parallel block generation with KV-injected target context; DSpark reuses this wholesale as its “expensive, parallel” half.
• Sequential correction lineage: conceptually adjacent to Domino’s CausalEncoder (Huang et al., 2026a) and to CRF-NAT/CTC-drafter’s idea of adding a sequential layer atop parallel hidden states — but DSpark’s local factorization (Equation 4) is deliberately simpler than a globally normalized CRF, trading some modeling power for exact, tractable per-token probabilities.
• Confidence estimation lineage: draws on SpecDec++ (Huang et al., 2024) and prior confidence-based adaptive-length work, but adds the analytical total-variation supervision (Equation 8) and the calibration step (STS) that most threshold-based prior work lacks.
• Scheduling lineage: related to goodput-oriented systems work (TurboSpec, Liu et al., 2024c) that optimizes speculative decoding as a system-level resource allocation problem, but DSpark is distinguished by deriving an exact greedy optimum with a formal non-anticipation proof, rather than a heuristic or learned bandit policy (contrast with Liu et al., 2026b’s bandit-style drafter selection).
• Tree-based alternatives: DDTree, TAPS, and JetSpec (Hu et al., 2026a; Ringel and Romano, 2026; Wang et al., 2026a) extend the draft chain into a verifiable tree structure rather than a linear block — a orthogonal axis of scaling that DSpark does not explore in this paper (DSpark uses chain-based drafting throughout, per Section 4.1), leaving tree-structured semi-autoregressive drafting as an open combination for future work.

Limitations and Boundary Conditions (My Assessment)

Beyond the paper’s own stated limitation (unrecoverable draft-side compute), a few boundary conditions are worth naming explicitly:

• The unimodality assumption is doing real work. The provable optimality of Algorithm 1 rests on $\Theta$ being unimodal in $B$. The production system works around this by allowing an unconstrained search plus a two-step-lagged causal barrier — but this means the theoretical optimality guarantee from Section 3.2.2 and the deployed algorithm in Section 5.2 are not quite the same algorithm; the deployed version is empirically effective but its optimality is no longer formally proven in the presence of a jagged $SPS(B)$.
• The confidence-scheduling gains are conditional on genuinely spare capacity. The paper’s own framing (Section 5.3) notes that in their specific deployment, effective batch size stays “well below the GPU’s compute-saturating threshold” due to KV-cache and traffic-pool limits — a favorable regime where throughput and per-user latency happen to be correlated rather than opposed. Operators running closer to compute saturation may see a less favorable trade-off between the two objectives.
• Markov-head boundary case. As discussed above, the default Markov head only fixes first-order collisions; multi-token semantic inconsistencies spanning 3+ positions are architecturally out of its reach, and the paper’s own ablation shows the RNN head (which could catch these) provides only marginal extra benefit in practice, suggesting such longer-range collisions may be rare in the tested domains but not necessarily in all deployment settings (e.g., highly structured formats with rigid multi-token idioms).
• Vocabulary size is a hidden multiplier on the Markov head’s cost. The complexity analysis above (Equation 15) shows the sequential correction’s FLOP cost scales linearly in vocabulary size $V$. DeepSeek-family models use a large but not extreme vocabulary; a deployment onto a target model with a substantially larger vocabulary (e.g., some multilingual or byte-level tokenizer schemes push $V$ well past $10^5$) would proportionally inflate the correction module’s cost, potentially eroding the “negligible overhead” property the production ablation reports — the paper’s cost measurements should be read as tied to DeepSeek-V4’s specific tokenizer, not as a vocabulary-agnostic guarantee.
• The scheduler’s guarantees are per-round, not per-conversation. Algorithm 1 optimizes system-wide throughput for the current decoding round in isolation; nothing in the formulation accounts for fairness or starvation across rounds (e.g., whether the same handful of low-confidence requests get their verification budget squeezed toward the MTP-1 floor round after round while high-confidence requests consistently win the greedy admission). The paper’s production telemetry (Figure 8) reports aggregate, load-bucketed behavior rather than per-request budget trajectories over time, so this is a real (if plausibly minor) omission for anyone deploying the scheduler under an SLA that promises individual users a minimum experience, not just a good aggregate number.

Critical Assessment: Weaknesses & Improvements

(a) Weaknesses and unconvincing/missing comparisons.

1. No apples-to-apples comparison against other confidence-scheduled or SLO-aware baselines in the online deployment. The production comparison (Section 5.4) is DSpark vs. MTP-1, a static, non-scheduled single-token drafter. The paper’s own Related Work section names several confidence-aware or goodput-optimizing schedulers (SpecDec++, AdaSpec, TurboSpec-style goodput optimization) that arguably form the more relevant online baseline — comparing against them (even offline, using the same static-threshold protocol used for the confidence-head validation in Section 4.3.3) would isolate how much of the 60-85% gain comes from the semi-autoregressive drafter itself versus the scheduler, since MTP-1 is weak on both axes simultaneously and conflates the two contributions.
2. The “nominal 661%/406%” numbers are reported at all, even with the caveat. The paper is admirably self-aware that these numbers overstate the real effect (MTP-1 “nearly collapses” at strict SLAs), but headline percentages this large will inevitably get quoted out of context in secondary reporting. A cleaner presentation would report only the matched-throughput comparison (60-85%/57-78%) as the primary claim and relegate the frontier-extension numbers to a qualitative “the baseline becomes infeasible here” statement rather than a specific multiplier.
3. No variance or confidence intervals anywhere in the offline results (Table 1) or the online production figures. Accepted length is reported as a single point estimate per benchmark; given that some benchmarks (AIME25, Arena-Hard) are small or high-variance by nature, it is not possible from the paper alone to judge whether, e.g., the 30.9% vs. 26.7% vs. 30.0% improvement pattern across model scales (4B/8B/14B) is a real trend or benchmark noise.
4. The “smoothly decaying, unimodal SPS(B)” assumption is acknowledged as false in practice (Section 5.2) but the paper never quantifies how far real hardware curves deviate from unimodality, nor does it report how much throughput is left on the table by the asynchronous, two-step-lagged approximation relative to a hypothetical oracle scheduler with perfect foresight. A quantified “optimality gap” for the production algorithm (vs. the theoretical Algorithm 1) would substantiate the claim that the practical adaptation is nearly as good as the provably-optimal version.
5. Only one draft-model configuration is deployed in production ($\gamma=5$, Markov head, 3 MoE layers). The rich offline ablation space (depth 1-5, $\gamma \in \{4,8,12,16\}$, Markov vs. RNN head) is never connected back to why these particular production hyperparameters were chosen over, say, a slightly larger $\gamma$ given that the offline ablations show DSpark’s relative advantage over DFlash grows with $\gamma$.

(b) Limitations understated or omitted.

1. The paper does not discuss what happens to scheduling quality when confidence-head calibration drifts over time (e.g., as the underlying model is updated, or as traffic distribution shifts from the STS calibration set) — STS is calibrated once on a held-out set, but production traffic composition (chat vs. code vs. math mix) is exactly the axis the paper shows drives large behavioral differences in acceptance rate, so a calibration set that no longer matches live traffic composition could silently degrade scheduler quality without any obvious failure signal.
2. The training data (Open-PerfectBlend) skews heavily toward math (39.4%) and code (38.9%) with comparatively little chat (17.6%) — yet chat is exactly the domain where the paper shows both the confidence head has the most pruning headroom (Figure 5) and where the drafter’s absolute accepted length is lowest (Table 1). It’s plausible more chat-domain training data specifically would move the needle on real-world (much more chat-heavy) production workloads more than further math/code-focused tuning.
3. The paper does not report the compute/parameter cost of the draft model itself (training FLOPs, additional GPU memory footprint of the co-deployed drafter, or its own serving cost as a fraction of the target model), which matters for any team trying to estimate the total cost of adopting this approach rather than just the speedup it produces.

(c) Concrete, actionable improvement suggestions.

1. Run an offline ablation that swaps only the scheduler (holding the DSpark drafter fixed) between: no scheduling (verify full block), static-threshold scheduling (as in Section 4.3.3), and the full hardware-aware scheduler — this would cleanly decompose how much of the 60-85% online gain is attributable to the drafter architecture versus the scheduler, information the current experiment design cannot separate.
2. Report a controlled “confidence drift” stress test: recalibrate STS on data from month $t$, then measure ECE and realized scheduler throughput on traffic from month $t+k$ for increasing $k$, to characterize how often recalibration is actually needed in production and whether the paper’s implicit assumption of a static calibration set is safe over realistic deployment timescales.
3. Add an explicit ablation on training-data domain mixture (e.g., a chat-heavy variant of Open-PerfectBlend) and report whether it closes the acceptance-rate gap between chat and math/code, directly testing hypothesis (b.2) above rather than leaving it as an open question.
4. Publish the empirical optimality gap of the deployed asynchronous scheduler (Section 5.2) against an oracle that has 1-step lookahead into $c_{k+1}$ (a version that is not deployable online but is computable offline for comparison), to substantiate that the 2-step-lag approximation costs little in practice rather than relying only on the qualitative argument that it “restores causality.”
5. Report the additional GPU memory and serving-cost overhead of co-deploying the draft model (backbone + sequential head + confidence head) alongside DeepSeek-V4, expressed as a percentage of the target model’s own footprint, so that adopters outside DeepSeek’s own infrastructure can estimate feasibility for their own serving stacks.
6. Publish the per-position ECE/AUC reliability breakdown (reproduced in this review’s Figure 5b) for all three evaluation domains, not only Alpaca — since this review’s own analysis above finds the pre-calibration ECE is non-monotonic across positions in a way the paper does not explain, a cross-domain version of this same table would clarify whether that non-monotonicity is a stable property of the confidence head or an artifact specific to Alpaca’s particular chat-style traffic.

Reproducibility Notes

The authors release DSpark checkpoints for DeepSeek-V4-Flash (preview) and DeepSeek-V4-Pro (preview), plus DeepSpec, an open-source, algorithm-driven training repository implementing Eagle3, DFlash, and DSpark under one framework — a genuinely useful contribution for reproducibility, since it means all three baselines compared in Table 1 were trained with a shared, inspectable codebase rather than each baseline being reimplemented independently (a common source of unfair baseline comparisons in the speculative-decoding literature that this paper explicitly avoids). The training recipe (Open-PerfectBlend, 10 epochs, position-weighted losses with the specific weights given in Equation 14) is fully specified in the paper, and the confidence-scheduling algorithm (Algorithm 1) is precise enough to reimplement directly, including the non-anticipation counterexample that any reimplementation should use as a unit test for correctness.

Estimating Your Own Speedup: A Back-of-Envelope Calculator

For a reader wondering “would this help my own serving stack,” Equation 1 ($L = (T_{draft}+T_{verify})/\tau$) doubles as a rough calculator once you have four numbers for your own setup: your target model’s plain per-token decode latency $L_0$ (i.e., $T_{verify}$ for $\gamma=0$, no drafting at all), your drafter’s per-round latency $T_{draft}$, your verification pass’s latency at your chosen block size $T_{verify}(\gamma)$, and an estimate of your expected accepted length $\tau(\gamma)$ at that block size (which you would need to measure empirically, e.g., via the offline accepted-length methodology in the “Experiments” section above, since it depends on how well-matched your drafter is to your target model).

Worked example using this review’s own earlier numbers. Recall the illustrative Equation 1 example: $T_{draft}=3$ ms, $T_{verify}=25$ ms at $\gamma=8$, giving $\tau=3.5$ (a DFlash-like parallel drafter with suffix decay) versus $\tau=5.5$ (a DSpark-like drafter with the decay suppressed). The relative speedup of adopting DSpark-style semi-autoregression over a plain parallel drafter, holding $T_{draft}$ and $T_{verify}$ fixed (a reasonable approximation given the complexity analysis above shows the sequential correction adds negligible latency), is simply the ratio of accepted lengths:

$$\frac{L_{DFlash}}{L_{DSpark}} = \frac{(T_{draft}+T_{verify})/\tau_{DFlash}}{(T_{draft}+T_{verify})/\tau_{DSpark}} = \frac{\tau_{DSpark}}{\tau_{DFlash}} = \frac{5.5}{3.5} \approx 1.57 \tag{16}$$

i.e., a 57% per-token latency reduction from the accepted-length improvement alone — in the same ballpark as the paper’s own reported 60-85% production gain, which additionally benefits from the confidence-scheduled verifier reducing the effective $T_{verify}$ under load (the second, independent lever this review’s Figure 3b table highlighted). The calculator’s key limitation, worth stating plainly: it assumes you can measure $\tau$ for your own model/domain pair, which in practice requires running the same offline accepted-length benchmark methodology (Section “Experiments: Offline Benchmarks” above) on your own workload — the 30-85% figures in this paper are specific to DeepSeek-V4 and its training/serving stack and should be treated as an upper-bound reference point, not a guaranteed transferable number, especially given the paper’s own domain-variance findings (math/code accepted length consistently higher than chat) suggest your result will depend heavily on your traffic mix.

Common Questions Answered

Does DSpark ever return a different output than plain autoregressive decoding from the target model? No — this is the entire point of speculative decoding’s rejection-sampling acceptance rule (Equation in Section “Speculative Decoding: Draft, Then Verify” above), and DSpark is explicit about preserving it at every layer: the sequential head produces exact softmax probabilities (Equation 4), the confidence head’s calibration (STS) is order-preserving so it cannot change which tokens get admitted relative to their true rank, and the scheduler’s non-anticipating property (proven via the Appendix A counterexample) ensures the choice of how many tokens to verify never depends on information that would leak the yet-unverified draft tokens’ identities. Every optimization in the paper is a latency/throughput optimization, not an approximation of the output distribution.

Why not just always verify the full draft block, given that verification is “nearly free” compared to a full autoregressive step? This is true for a single request in isolation, but false at the system level under concurrency (see the “Continuous Batching” prerequisite section above): every additional verified token effectively adds a slot to the shared batch, and GPU throughput per step degrades as batch size grows. Table entries in Figure 8 make this concrete — the scheduler shrinks verification budget from ~5-6 tokens/request at low load down toward MTP-1’s static 2 tokens as concurrency approaches saturation, precisely because “nearly free” stops being true once the GPU is the bottleneck rather than any individual request.

Could you swap in a completely different parallel backbone instead of DFlash? Nothing in the semi-autoregressive formulation (Equation 4) is DFlash-specific — the recipe only requires a backbone that produces per-position hidden states $h_k$ and base logits $U_k$ in a single forward pass, which any parallel drafter (e.g., Medusa-style multi-head prediction) could supply. The paper chooses DFlash because it was, at the time, state-of-the-art among parallel drafters, but the sequential correction head and the confidence-scheduled verifier are best understood as backbone-agnostic add-ons.

Is the hardware-aware scheduler specific to DeepSeek’s serving stack, or portable to other inference engines (vLLM, SGLang, TensorRT-LLM)? The core algorithm (Algorithm 1) only needs two engine-provided primitives: (1) per-position confidence estimates from the drafter, and (2) a profiled throughput-vs-batch-size lookup table $SPS(B)$, which any serving engine can produce via offline benchmarking regardless of its internal kernel implementation. The production-specific parts (Section 5.2’s two-step-lag adaptation for CUDA graph replay, and Section 5.3’s flattened variable-length kernel routing) are more engine-specific, since they interact directly with how a given engine implements continuous batching and graph capture — a team adopting this on a different engine would likely need to re-derive the equivalent of these two adaptations for their own kernel and scheduling stack rather than porting the DeepSeek-specific code directly.

Why does the paper bother with the Markov head at all, given the RNN head is strictly more expressive? Because expressiveness is not free even when its latency cost is negligible (Section 4.3.2 shows sequential-loop overhead is small for both): the RNN head is a more complex module to implement correctly, verify, and maintain in a production training/serving pipeline, and the measured accuracy gain over the Markov head is only marginal, concentrated at long proposal lengths that may not represent the majority of real production traffic. This is a case where the paper explicitly chooses simplicity over a small amount of extra headroom — a reasonable trade for a production system, though it does mean the RNN head’s larger potential (e.g., in domains with longer-range intra-block dependencies not represented in Open-PerfectBlend) remains largely unexplored in the deployed system.

What happens when two candidates tie exactly on marginal value $a_{r,j}$? The formal proof sketch above (Lemma 2) only requires that the sum of admitted values be maximized, not that any particular tie be broken a specific way — swapping two tied candidates leaves $\tau^*(B)$ and hence $\Theta(B)$ unchanged at every $B$. The one place tie-breaking could matter operationally is exactly at the batch size where $\Theta$ transitions from increasing to decreasing; a deterministic tie-break (e.g., lowest request ID first) is sufficient to make the algorithm reproducible without affecting the guaranteed optimal value of $\Theta_{best}$ itself.

How would $SPS(B)$ actually be measured in a production engine, and does it ever go stale? $SPS(B)$ is profiled once, offline, at engine startup by benchmarking decode-step wall-clock time across a sweep of batch sizes on the target hardware and kernel configuration, producing the lookup table Algorithm 1 consults. It is a property of the serving stack (GPU model, kernel implementation, tensor/pipeline-parallel layout), not of any particular request, so it does not need per-request updates — but it would need re-profiling after a kernel upgrade, a hardware swap, or a change in parallelism configuration, none of which the paper discusses as an operational concern.

Does any of this change under greedy (temperature-0) decoding? The acceptance-rate identity derived above (Equation 8c) is built on stochastic rejection sampling and is not directly meaningful at temperature 0, where “acceptance” degenerates into an exact-match check between the draft token and the target’s single argmax token. The general speculative-decoding framework still applies in this limit (Leviathan et al., 2023 cover it as a special case), but DSpark’s analytical confidence label $c_k^*$ would correspondingly degenerate from a smooth distance-based quantity into a harder 0/1 indicator of argmax agreement — a regime the paper does not analyze separately, and one where the STS calibration story (built around smoothly varying probabilities) would need independent verification before being trusted.

Formula Reference: Every Equation in This Review, Cross-Referenced

Because this review spans drafting, calibration, scheduling, and training, and reuses several equations across multiple later sections (Equation 8 alone is invoked in the confidence head, the calibration discussion, the FAQ, and the back-of-envelope calculator), it is useful to have one consolidated index rather than hunting back through the text.

Eq.	What it defines	First introduced in	Reused in
1	Per-token latency $L=(T_{draft}+T_{verify})/\tau$	“Speculative Decoding: Draft, Then Verify”	Back-of-envelope calculator, Conclusion
2	DFlash context feature extraction $H_{ctx}$	“Recap of the DFlash Parallel Backbone”	—
3	DFlash context injection into K/V	”Recap of the DFlash Parallel Backbone”	—
4	Autoregressive factorization over base logits + bias	”The Sequential Correction”	Pseudocode Step 5, FAQ
5	Low-rank Markov bigram bias	”Instantiation 1: The Markov Head”	Complexity analysis (Eq. 15)
6	RNN head recurrent update	”Instantiation 2: The RNN Head”	—
7	Raw confidence head $c_k$	“The Confidence Head”	Pseudocode Step 6
8 / 8a-c	Analytical acceptance label $c_k^*$ and its TV-distance derivation	”The Confidence Head” / derivation subsection	Training loss (Eq. 13), FAQ, calculator
9-10	Non-anticipation counterexample throughputs and output distribution	”The Non-Anticipating Property”	—
11-14	Training losses ($\mathcal{L}_{ce}, \mathcal{L}_{tv}, \mathcal{L}_{conf}$, combined $\mathcal{L}$)	“Training Objective”	—
15	Sequential-head FLOP cost	”A Complexity Analysis”	FAQ (vocabulary scaling limitation)
16	Ratio-based speedup calculator	”Estimating Your Own Speedup”	Conclusion

Putting It All Together: The Full Decode Loop as Pseudocode

The paper presents drafting, calibration, and scheduling in separate subsections; it is useful to see them fused into one end-to-end loop, since that is what actually executes on the GPU each round:

Algorithm: DSpark End-to-End Decoding Round (per batch of R requests)
Require: anchor tokens x0_1..x0_R (each request's previous bonus token),
         frozen target model M_t, DSpark draft model (backbone + seq head + conf head),
         calibrated STS temperatures, profiled SPS(B) table, capacity estimate K
         from confidence outputs two steps prior

1.  For each request r in 1..R (batched together in one parallel forward pass):
2.      Run parallel backbone on anchor x0_r -> base logits U_1..U_gamma, hidden states h_1..h_gamma
3.      For k = 1 to gamma:                                   # lightweight sequential loop
4.          Compute bias B_k from Markov head (Eq. 5) or RNN head (Eq. 6)
5.          Sample x_{r,k} ~ softmax(U_k + B_k)                # Eq. 4
6.          Compute raw confidence c_{r,k} (Eq. 7); apply STS calibration
7.      Compute cumulative survival a_{r,j} = prod_{i<=j} c_{r,i} for j = 1..gamma
8.  Pool all (r,j) candidates across the R requests; sort descending by a_{r,j}
9.  Greedily admit candidates while Theta = tau* x SPS(B) keeps improving,
    using the capacity ceiling K derived from confidences two steps prior (Sec. 5.2)
10. Assemble the flattened, variable-length verification batch (marker tensor encodes structure)
11. Target model M_t verifies the scheduled prefixes in one parallel forward pass
12. For each request r: accept longest prefix consistent with M_t's own distribution;
    resample one corrected token at the first rejection (or after full acceptance, a bonus token)
13. New anchor x0_r <- last accepted/resampled token; go to step 1 for the next round

This full loop makes explicit something that is easy to miss reading the paper section by section: steps 2-7 (drafting + confidence estimation) happen entirely on the draft model’s compute path and are cheap by construction, step 8-10 (scheduling + batch assembly) is pure bookkeeping with no matrix multiplication at all, and only steps 11-12 touch the expensive target model — meaning the entire mechanism that makes DSpark “smart” (semi-autoregressive correction, calibration, scheduling) adds negligible FLOPs to the round; it changes which and how many tokens reach the expensive step, not how expensive that step itself is per token.

A Three-Request Numeric Trace Through the Whole Loop

The two-request scheduler example earlier isolated Steps 8-9 in isolation. It is worth running one complete trace through every numbered step above, with three concurrent requests of different character (a confident math request, a middling code request, and a diffuse open-ended chat request), to see how the whole pipeline’s pieces compose in practice.

Setup. $R=3$, $\gamma=3$. Suppose the confidence head (after STS calibration) produces:

• Request 1 (math-like, high confidence): $c_{1,1}=0.95, c_{1,2}=0.85, c_{1,3}=0.70$
• Request 2 (code-like, moderate confidence): $c_{2,1}=0.80, c_{2,2}=0.55, c_{2,3}=0.30$
• Request 3 (chat-like, low confidence, steep decay): $c_{3,1}=0.65, c_{3,2}=0.25, c_{3,3}=0.10$

Step 7 — cumulative survival probabilities $a_{r,j} = \prod_{i\le j} c_{r,i}$:

$r$	$a_{r,1}$	$a_{r,2}$	$a_{r,3}$
1 (math)	0.950	0.808	0.565
2 (code)	0.800	0.440	0.132
3 (chat)	0.650	0.163	0.016

Step 8 — pool and sort descending. Pooling all nine $(r,j)$ candidates and sorting by survival probability $a_{r,j}$ gives: $(1,1){:}0.950 > (1,2){:}0.808 > (2,1){:}0.800 > (3,1){:}0.650 > (1,3){:}0.565 > (2,2){:}0.440 > (3,2){:}0.163 > (2,3){:}0.132 > (3,3){:}0.016$.

Step 9 — greedy admission. Suppose the profiled capacity curve is $SPS(3)=1.0, SPS(4)=0.85, SPS(5)=0.70, SPS(6)=0.60, SPS(7)=0.42, SPS(8)=0.30, SPS(9)=0.20, SPS(10)=0.12, SPS(11)=0.07, SPS(12)=0.04$ (a smoothly decaying, unimodal-with-respect-to-$\Theta$ curve, chosen for illustration). Starting from $B=3$ (one anchor slot per request), $\tau^*=3$, $\Theta_{best}=3\times SPS(3)=3.00$:

Admit	$B$	$\tau^*$	$SPS(B)$	$\Theta$	Decision
$(1,1)$	4	3.950	0.85	3.358	accept (new best)
$(1,2)$	5	4.758	0.70	3.331	below best → break

The causal early-stopping scheduler halts here: $\ell_1^*=1, \ell_2^*=0, \ell_3^*=0$, with $\Theta_{best}=3.358$. Continuing the table purely for intuition (not executed by the causal algorithm) confirms the curve keeps falling: admitting $(2,1)$ next would give $B=6, \tau^*=5.558, \Theta=5.558\times0.60=3.335$; admitting $(3,1)$ after that gives $B=7,\tau^*=6.208,\Theta=6.208\times0.42=2.607$ — monotonically worse, so the early stop at $B=4$ is in fact the global optimum on this particular curve, consistent with the proof sketch above.

Interpretation. Only the math request’s first draft token gets verified this round beyond the guaranteed anchor — despite the code request’s own first-position confidence (0.800) looking individually reasonable, and even the chat request’s first position (0.650) being non-trivial. This is the scheduler correctly recognizing that, given the shared cost of growing the batch (captured by the falling $SPS(B)$ curve), the system-wide payoff of spending one more verification slot on request 1’s second token is not as good as it first appears once you account for what verifying it does to $SPS(B)$ for everyone. It also illustrates a subtlety worth flagging explicitly: the greedy scan can admit a second token from one request (e.g., $(1,2)$) before ever considering a competing request’s first token, precisely because the ranking is by absolute marginal value $a_{r,j}$, not by request or by position — a within-request-second-token candidate can legitimately outrank a between-request first-token candidate if its survival probability is higher, which is exactly what nearly happened here between $(1,2){:}0.808$ and $(2,1){:}0.800$ (they are close enough that a slightly different confidence estimate would flip their relative order, a good illustration of why calibration accuracy, not just ranking accuracy, matters for getting this comparison right at the margin).

Steps 10-13. The scheduler hands off $\ell^* = (1, 0, 0)$ to the batch-assembly stage, which flattens the four total tokens to verify (one anchor slot each for requests 2 and 3, one anchor plus one draft slot for request 1) into a single physically homogeneous stream with a marker tensor recording which tokens belong to which request (Step 10); the target model verifies this batch in one parallel pass (Step 11); each request accepts its longest valid prefix and receives a bonus/resampled token from the target’s own distribution (Step 12); and the newly accepted tokens become next round’s anchors (Step 13) — request 1 potentially advancing by up to 2 tokens this round (1 draft + 1 bonus) while requests 2 and 3 advance by exactly 1 token each (the bonus token alone), exactly as plain one-token decoding would, at zero cost to correctness for either of them.

Design Decisions at a Glance

Summarizing the “why this, not the obvious alternative” reasoning threaded through the sections above into one table:

Decision	What DSpark chose	The obvious alternative	Why the alternative falls short
Drafter structure	Semi-autoregressive (parallel + tiny sequential head)	Pure parallel, or pure autoregressive	Pure parallel suffers suffix decay; pure autoregressive pays $O(\gamma)$ drafting cost and forces shallow nets
Sequential correction rank	Low-rank ($r=256$) Markov bigram bias	Full $V \times V$ transition matrix	Full matrix is memory- and compute-prohibitive for $V\approx10^5$
Default correction module	Markov head (1-step memory)	RNN head (full intra-block memory)	RNN gives only marginal extra gain at higher implementation/maintenance cost
Confidence supervision	Analytical TV-distance label $c_k^*$ (Eq. 8)	Empirical accept/reject binary label	Analytical label is exact and low-variance; empirical label needs many samples per position
Confidence calibration	Sequential Temperature Scaling (order-preserving)	Uncalibrated raw sigmoid outputs	Raw outputs are overconfident (ECE 3-8%), which would corrupt the scheduler’s throughput arithmetic
Verification-length policy	Global greedy scheduler over pooled per-token marginal values	Independent static/per-request threshold	Static thresholds ignore system load and cannot compare marginal value across requests
Production scheduling timing	Two-step-lagged capacity estimate, current-step ranking	Fully synchronous per-step scheduling	Synchronous scheduling stalls the GPU pipeline and breaks CUDA graph replay / ZOS
Kernel execution	Flattened variable-length tokens + marker tensor	Padded fixed-length batches	Padding wastes compute and causes uneven GPU utilization under dynamic lengths

Conclusion

DSpark’s core insight is refreshingly simple once stated: a parallel drafter’s independence assumption is the quality bottleneck, and a static verification length is the system bottleneck, but neither bottleneck needs to be solved with a heavyweight fix. A tiny sequential correction module (a rank-256 bigram bias, in the default configuration) recovers most of the coherence lost to parallel generation, and a provably-lossless greedy scheduler — built on the simple but non-obvious observation that per-token marginal value is comparable across requests — recovers most of the throughput lost to blind fixed-length verification. What elevates this from “a clever idea” to a paper worth reading closely is the rigor applied to the scheduler’s correctness (the non-anticipating property, proven with an explicit counterexample) and the honesty in the production section about where the clean theory (Algorithm 1) had to bend to survive contact with real, jagged hardware capacity curves and CUDA-graph-based serving engines. The 60-85% production speedup is a strong result, but the more durable contribution is arguably the general recipe: whenever a component of a serving pipeline can emit a calibrated, well-defined confidence score, that score can potentially be turned into a provably-safe, load-aware scheduling decision rather than a hand-tuned threshold — a pattern likely to recur well beyond speculative decoding.