TreeWY: Removing the Memory Wall in Speculative Decoding for Gated DeltaNet Hybrids

Review date: 2026-09-05 Paper reviewed: TreeWY: Speculative Verification for Gated DeltaNet Hybrids Paper authors: Sneha Murthy Ghantasala (Thomson Reuters) arXiv: 2608.20961 Venue/Status: arXiv preprint, submitted 21 Aug 2026

1. What Problem Is This Paper Actually Solving?

Modern open-weight LLMs are increasingly hybrids: instead of putting softmax self-attention in every layer, they mix in linear-attention layers — most commonly Gated DeltaNet (GDN) blocks — that replace the classic growing key/value (KV) cache with a single fixed-size recurrent state matrix per head. This is attractive for exactly one reason: GDN’s memory footprint per sequence is constant in context length, whereas a softmax layer’s KV cache grows linearly and eventually dominates GPU memory at long context. Recent large open models (Qwen3.5 being the example this paper studies) ship in a roughly 3:1 GDN-to-softmax ratio specifically to buy that memory win.

The catch is that this same property — a single fixed-size state that has already absorbed the entire prefix — is exactly what makes speculative decoding awkward for GDN layers. Speculative decoding works by having a cheap “drafter” propose several candidate tokens ahead of the “real” model, and then verifying all of them in one parallel forward pass through the target model; whichever prefix of draft tokens the target would have generated itself gets accepted, and the model rolls back to that point before continuing. For an ordinary softmax layer, rollback is nearly free: the KV cache is just a list, you append draft entries during the speculative pass and discard the rejected ones by moving a pointer. But a GDN layer’s recurrent state has already been overwritten by the whole draft sequence by the time verification finishes — there is no way to recover the state as it looked halfway through the draft without having saved it. The naive fix, which the paper calls full-state snapshotting and which is the default in both vLLM and SGLang today, is to save a full copy of the recurrent state at every single draft position, so you can jump back to whichever one turns out to be accepted. For a chain of $k$ draft tokens that’s $k+1$ full state copies; for a tree of $N$ candidate nodes (where a wide tree proposes multiple alternatives per position to raise the acceptance rate), it’s $N+1$ copies, and — crucially — those snapshots cannot be shared across different branches of the tree, because each branch’s history diverges. Snapshot memory therefore scales directly with how aggressively you speculate, which caps how wide a draft tree you can afford and eats into exactly the memory budget that GDN was supposed to free up.

TreeWY’s contribution is a way to make this problem disappear rather than trading it off. The paper observes that the gated delta rule that defines GDN’s recurrence can be re-derived, algebraically, as a special case of the WY transform — a classical numerical linear algebra trick (originally from Householder QR factorization, and already used in the DeltaNet literature to parallelize training over sequence length) that expresses a chain of rank-1 state updates as a single triangular linear system. TreeWY extends this idea from a chain to a full draft tree, and from training-time parallelization to verification-time parallelization: every node in a speculative draft tree, no matter how wide, has its “would-have-been” recurrent-state contribution computed by one forward substitution over a strictly lower-triangular system, and when verification finally decides which node was accepted, the paper reconstructs only that one state directly from the solved values — never having stored, and never needing to store, the other $N$ candidate states along the way. The practical payoff, measured by implementing this inside vLLM and serving two scales of a real hybrid model family (Qwen3.5 35B-A3B and 397B-A17B) on B200 GPUs: 2–3× less peak KV/state memory at matched load, which translates into up to 1.49× higher throughput and roughly 30–40× lower p99 time-to-first-token specifically in the regime where the baseline’s memory pool was the bottleneck, at the cost of a small (a few percent) throughput hit in the regime where memory wasn’t binding anyway.

2. Prerequisites: What You Need to Understand First

2.1 Autoregressive decoding and why it’s slow

A Transformer language model, given a prompt, generates text one token at a time: it runs a forward pass, samples the next token from the output distribution, appends that token to the sequence, and repeats. Each forward pass at generation time is memory-bandwidth bound, not compute bound — the model has to read every one of its weights (and every entry currently in the KV cache) from GPU HBM into on-chip memory to produce just one new token, so most of the GPU’s arithmetic capacity sits idle. This is the fundamental inefficiency that speculative decoding exploits: if you could somehow produce several tokens’ worth of useful work per weight-read instead of one, you’d get proportionally more throughput for free.

2.2 Speculative decoding, formally

Speculative decoding (Leviathan et al. 2023; Chen et al. 2023) fixes the idle-compute problem by decoupling “propose” from “verify.” A cheap drafter — a small separate model, or a lightweight extra “head” attached to the target model (e.g., an MTP — multi-token-prediction — head) — proposes $k$ candidate tokens autoregressively, which is cheap because the drafter is small. The target model then runs one single forward pass over all $k$ candidate positions simultaneously (this is where the idle compute gets used), computing what its own output distribution would have been at each position. Because this verification is done in parallel, it costs roughly the same wall-clock time as generating one token normally, but produces up to $k$ tokens if every candidate is accepted.

The acceptance rule that makes this exactly match the target model’s own sampling distribution (this is what “lossless” means here) is:

$$p_{\text{accept}}(x_i) = \min\left(1, \frac{p_{\text{target}}(x_i \mid x_{<i})}{p_{\text{draft}}(x_i \mid x_{<i})}\right), \tag{1}$$

evaluated position by position from $i=1$ until the first rejection, at which point a corrected residual distribution is sampled to replace the rejected token, and everything after it is discarded. Crucially, the model’s output distribution is provably identical to what greedy/sampled decoding without speculation would have produced — the speedup is “free” in the sense that it doesn’t trade off quality for speed, only wasted-compute for used-compute.

A refinement of this idea is tree-structured drafting (e.g., Medusa, and the $N$-node trees this paper studies): instead of proposing one linear chain of $k$ candidates, the drafter proposes a tree, where a given position can branch into several alternative next tokens, each of which can itself branch further. Verifying a tree lets the target model accept whichever root-to-node path through the tree it agrees with, which on average accepts more tokens per verification round than a single chain of the same depth, because the tree gives the drafter more chances to guess right at each step. The tradeoff is that a tree of $N$ nodes costs more to verify (attention/state computation over $N$ positions rather than $k$), and — as this paper is entirely about — costs more to store per node if each node needs its own saved state.

2.3 KV cache attention (the “normal” case)

For an ordinary softmax-attention layer, at each generation step the model computes new key ($k_t$) and value ($v_t$) vectors for the current token and appends them to a growing cache: $K, V \in \mathbb{R}^{T \times d}$ after $T$ tokens. Attention over the whole prefix is:

$$\mathrm{Attn}(q_t, K_{\le t}, V_{\le t}) = \mathrm{softmax}\!\left(\frac{q_t K_{\le t}^\top}{\sqrt{d}}\right) V_{\le t}. \tag{2}$$

Under speculative decoding, this cache is append-and-truncate friendly: draft tokens’ keys/values get appended during the verification pass, and if a draft token is rejected, you simply move a pointer back to drop everything after the accepted prefix — no recomputation, no extra storage, and (for a tree) branches sharing a common prefix can literally share the same cache entries for that prefix. This is why the paper says “softmax layers stay cheap” under speculation: the data structure itself is naturally rollback-friendly.

2.4 Linear attention and the Gated DeltaNet (GDN) recurrence

A linear-attention layer replaces the growing KV cache with a single fixed-size matrix state $S \in \mathbb{R}^{d_v \times d_k}$ per head that is updated recurrently as each token arrives, and read out via a matrix-vector product rather than a softmax over the whole history. The specific recurrence this paper is built around — the gated delta rule (Yang, Kautz & Hatamizadeh; used in Gated DeltaNet, ICLR 2025) — is:

$$S_t = \alpha_t S_{t-1}\left(I - \beta_t k_t k_t^\top\right) + \beta_t v_t k_t^\top, \qquad o_t = S_t q_t, \tag{3}$$

where $k_t, v_t, q_t$ are the usual key/value/query vectors for token $t$, $\alpha_t \in (0,1)$ is a scalar decay gate (how much of the old state survives), and $\beta_t \in (0,1)$ is a write strength. It helps to unpack this term by term:

• $\alpha_t S_{t-1}$: decay the entire previous state by a scalar factor before doing anything else. This is what lets old information fade rather than accumulate forever — necessary for a fixed-size state to remain informative indefinitely, since without decay a fixed-size summary would eventually saturate.
• $-\alpha_t \beta_t S_{t-1} k_t k_t^\top$: the “delta” correction. Before writing anything new, the layer removes whatever the (decayed) state already predicts for the incoming key $k_t$ — i.e., it computes $S_{t-1} k_t$ (a “guess” at what value should be associated with $k_t$ based on everything already stored) and subtracts a $\beta_t$-weighted version of the outer product this guess would produce. This is exactly the delta rule from classical associative-memory learning: instead of blindly writing new information on top of old, you write only the error between what the memory already predicts and what should actually be there. This is what prevents the state from being dominated by whichever key vector direction appears most often (a well-known failure mode of naive linear attention without this correction).
• $+\beta_t v_t k_t^\top$: after removing the old prediction, write the actual new value $v_t$ associated with key $k_t$, scaled by the same write strength $\beta_t$.
• $o_t = S_t q_t$: the read-out is a simple matrix-vector product against the current query — no softmax, no normalization, $O(d_v d_k)$ per token instead of $O(T d)$.

The crucial structural fact for everything that follows is that the transition operator $T_t = \alpha_t\left(I - \beta_t k_t k_t^\top\right)$ is a scalar decay times a rank-1 correction, and it does not commute across timesteps (unlike, say, a pure scalar-decay state-space model such as Mamba2, where the transition really is just a scalar and the whole recurrence collapses into a simple cumulative product/sum). This non-commutativity is exactly why techniques built for Mamba2-style models (like the paper’s cited prior work STree) don’t directly transfer to GDN — you can’t just turn the recurrence into a cumulative sum, because matrix multiplication order matters here.

2.5 Why GDN breaks speculative-decoding rollback

Under normal (non-speculative) decoding, GDN’s memory story is great: one state block per sequence, constant regardless of context length (30 MiB for the 35B model studied here, 90 MiB for the 397B model, versus a softmax KV cache that grows to 0.6–0.9 GiB at 32K context on the same models). But speculative verification runs one parallel forward pass over all $k$ (or $N$, for a tree) draft positions at once, which means the recurrence in Eq. (3) has to be evaluated all the way to the end of the draft sequence in that single pass, before the accept/reject decision (Eq. 1) has even been made. If the target model only accepts, say, the first 2 of 5 draft tokens, you need the state as it was after token 2 to continue decoding from there — but the forward pass has already computed the state all the way through token 5, overwriting whatever intermediate values existed.

The default fix in production systems today (vLLM, SGLang) is full-state snapshotting: save a complete copy of $S$ at every draft position during the pass, so that whichever position ends up being the accepted one, you can just index back into its saved snapshot. For a chain of $k=3$ draft tokens this is $k+1=4$ full state copies per sequence per GDN layer — a $4\times$ memory multiplier over the single committed state, i.e. 120 MiB/seq at 35B and 360 MiB/seq at 397B just for speculation bookkeeping. And critically, these snapshots are not shareable across tree branches: if the draft is a tree rather than a chain, every branch’s history diverges from wherever it forked off, so each of the $N$ nodes in an $N$-node tree needs its own full snapshot, with no possibility of reuse even for nodes that share a common ancestor prefix. This is the mechanism this paper set out to remove.

2.6 The WY transform (why it’s the right tool here)

The WY transform is a classical technique from numerical linear algebra (originally devised for representing a product of Householder reflections used in QR factorization as a single low-rank update, rather than as a literal sequence of matrix multiplications). Its relevance to linear attention was established by prior work on the plain (ungated) DeltaNet (Yang et al., “Parallelizing Linear Transformers with the Delta Rule over Sequence Length”), which showed that a chain of rank-1 delta-rule updates like Eq. (3) can be re-expressed as solving a single triangular linear system for a set of “effective” or “pseudo” values, rather than by literally recomputing the recurrence step by step. The reason this works is structural: each step’s update only depends on quantities computed at earlier steps (this is what “triangular” means here), so the whole chain of updates can be unrolled into one linear system where the unknowns are the effective per-step contributions, and solved in one shot via forward substitution rather than $T$ sequential recurrence steps. That prior work used this purely to speed up training (where you want to parallelize across the full sequence length instead of running the recurrence step-by-step, which is slow on a GPU). TreeWY’s contribution is to notice that the exact same structural trick — a chain of dependencies that only look backward — is also present in the speculative-verification problem, just with the “chain” replaced by a tree, and to work out the (non-trivial, because trees aren’t chains) linear-algebra needed to make it work for a tree rather than a chain, and for the gated delta rule specifically (which has the non-commuting decay-times-rank-1-correction structure described in §2.4) rather than the plain ungated version or a pure scalar-decay model like Mamba2.

2.7 Prior approaches to this exact problem, and why none of them quite close it

Before getting to TreeWY’s method, it’s worth being precise about the landscape it sits in, because the paper is explicit that it is not the only group to have noticed this gap:

• Full-state snapshotting (the incumbent, shipped in vLLM and SGLang today) is what was just described in §2.5: correct, simple, and the reason this paper exists. Its failure mode is purely a memory one — it does not compute anything wrong, it just cannot afford to keep $N+1$ full states around for a wide tree.
• ReplaySSM (a concurrent vLLM RFC and TensorRT-LLM PR, also covering Mamba2) keeps a checkpoint plus a short cached history and defers materializing the state until a periodic flush rather than snapshotting at every draft position. Structurally, inside its own verify window it solves the same triangular system TreeWY does — the distinction is when the state gets written to memory, not how it’s computed. Its published/linked implementation is chain-only, not tree-structured.
• STree (prior work by the same broader research area) verifies a tree, but only for Mamba2-style state-space models, whose transition operator is a bare scalar decay (no rank-1 correction). That means a whole chain of transitions collapses into a simple cumulative product, which is why STree’s tree extension is comparatively easy — GDN’s transition does not commute across timesteps (§2.4), so that trick does not carry over.
• Bole (concurrent work, same month, targeting SGLang) is the closest architectural relative: it also rewrites a linear-attention recurrence into a tree-structured closed form so a whole draft tree verifies in parallel, and reports large transient-memory and verification-speed wins. The TreeWY paper is candid that it has not run a head-to-head against Bole because Bole’s paper does not link to a code release — the two were arrived at independently, on different serving stacks, in the same few weeks.

The throughline: everyone agrees the recurrence itself can be turned into a triangular solve (this is not new — it’s DeltaNet’s WY transform, originally built for training-time parallelism, see §2.6). What’s contested is when to materialize state (ReplaySSM), whether the transform survives moving from a scalar transition to GDN’s rank-1-corrected one (STree does not attempt this; TreeWY and Bole both do, independently), and whether it generalizes to a tree rather than just a chain (only TreeWY and Bole claim this for GDN specifically). TreeWY’s specific claimed contribution, narrowly stated, is: extending the chain WY transform to a tree, for the gated (not plain, not Mamba2-scalar) delta rule, implemented and benchmarked inside vLLM.

3. Method: The Tree-Structured WY Transform

3.1 Step 1 — Rewriting the recurrence as decayed additive attention with a “pseudo-value”

The first move is purely algebraic: expand the gated delta rule of Eq. (3) and regroup terms. Starting from

$$S_t = \alpha_t S_{t-1}\left(I - \beta_t k_t k_t^\top\right) + \beta_t v_t k_t^\top,$$

distribute $\alpha_t S_{t-1}$ over the two terms inside the parentheses:

$$S_t = \alpha_t S_{t-1} - \alpha_t\beta_t S_{t-1} k_t k_t^\top + \beta_t v_t k_t^\top.$$

Now group the two terms that both carry a $k_t^\top$ on the right:

$$S_t = \alpha_t S_{t-1} + \left(\beta_t v_t - \alpha_t\beta_t S_{t-1} k_t\right) k_t^\top = \alpha_t S_{t-1} + \tilde v_t k_t^\top, \qquad \tilde v_t \triangleq \beta_t\left(v_t - \alpha_t S_{t-1} k_t\right). \tag{4}$$

This single rewrite is the crux of the whole method, so it is worth pausing on the intuition. Read $S_{t-1}k_t$ as “what the state, before this update, already predicts as the value associated with key $k_t$.” Then $\tilde v_t$ is not the raw incoming value $v_t$; it is $\beta_t$ times the residual between the real value and the state’s own prediction. In other words, Eq. (4) says the gated delta rule is secretly just decayed additive attention ($S_t = \alpha_t S_{t-1} + (\text{something}) k_t^\top$, structurally identical to the very simplest possible linear-attention recurrence) as long as you write the correct pseudo-value $\tilde v_t$ instead of the raw value. All of the delta-rule’s special “don’t overwrite what’s already there” behavior gets absorbed entirely into how $\tilde v_t$ is computed — none of it survives into the shape of the recurrence itself.

Why does this matter for speculative decoding? Because a chain of additive updates ($S_t = \alpha_t S_{t-1} + \tilde v_t k_t^\top$, repeated) unrolls into a closed form with no recurrence at all:

$$S_t = \left(\prod_{j=1}^{t} \alpha_j\right) S_0 + \sum_{i=1}^{t} \left(\prod_{j=i+1}^{t}\alpha_j\right) \tilde v_i k_i^\top = g_t S_0 + \sum_{i=1}^t \frac{g_t}{g_i}\, \tilde v_i k_i^\top, \qquad g_t \triangleq \prod_{j=1}^t \alpha_j. \tag{5}$$

Every state $S_t$ is now just a weighted sum over the pseudo-values $\tilde v_1, \dots, \tilde v_t$ computed so far, with weights that are cumulative-decay ratios $g_t/g_i$ that depend only on the scalar gates $\alpha_j$ — no matrix products in the exponent, unlike the raw recurrence. If you can get your hands on all the $\tilde v_i$ without walking the recurrence step by step, you can read out any $S_t$ (or any downstream output $o_t = S_t q_t$) directly from Eq. (5) — which is exactly what you need for verification, where you want the state at whichever node ends up accepted, not necessarily the last one computed.

The catch, and the reason this isn’t trivial, is that $\tilde v_t$ itself depends on $S_{t-1}$ (see the definition in Eq. 4) — so you seem to need the recurrence just to compute the pseudo-values in the first place, which looks circular. Section 3.2 shows how the paper breaks this circularity: rather than computing each $\tilde v_t$ sequentially, it sets up a single linear system whose solution simultaneously gives every $\tilde v_t$ at once.

3.2 Step 2 — One linear system for a whole draft tree

Algorithm 1 — Building and solving the tree verification system (informal pseudocode)

Input: committed state S0, draft tree nodes 1..N laid out in DFS pre-order
       (so every node's parent has a strictly smaller index),
       per-node (k_t, v_t, q_t, alpha_t, beta_t)
Output: pseudo-value matrix V_tilde in R^{N x d_v}, used for both
        per-node outputs and eventual state reconstruction

1. For every node t = 1..N, compute the cumulative decay g_t = prod_{j: j is an
   ancestor of t, inclusive} alpha_j        # a scalar per node, cheap prefix product
2. Build the strictly-lower-triangular ancestor-gated Gram matrix G:
     G[t, i] = (g_t / g_i) * beta_t * (k_t^T k_i)   if i is a strict ancestor of t
     G[t, i] = 0                                     otherwise (including i = t)
3. Build the right-hand side R:
     R[t] = beta_t * v_t  -  beta_t * g_t * (S0^T k_t)     # the "what S0 alone predicts" term
4. Solve the triangular system  (I + diag(beta) G) @ V_tilde = R   for V_tilde
   # strictly lower-triangular G means this is one forward substitution,
   # i.e. O(N) sequential steps of small matrix-vector work, not O(N) full
   # state recomputations - this is the actual computational win
5. Every node t's attention-style output can now be read out directly from
   V_tilde and g_t via the closed form of Eq. (5), with no per-node state ever
   materialized.

Unpacking why this pseudocode is correct: the key structural fact is that node $t$‘s pseudo-value $\tilde v_t$ only depends on $S_{t-1}$, and by Eq. (5), $S_{t-1}$ itself only depends on the pseudo-values of $t$‘s ancestors in the tree (not siblings, not descendants — this is what makes it a tree problem rather than requiring the model to consider all $N^2$ node pairs). Laying nodes out in DFS pre-order guarantees every ancestor has a strictly smaller index than its descendants, which is precisely the condition that makes the resulting matrix equation in Eq. (2) of the paper (reproduced as step 4 above) strictly lower-triangular — the unknowns $\tilde v_t$ for larger $t$ only ever depend on unknowns for smaller $t$, never the reverse. That triangularity is what turns “solve for $N$ mutually-dependent unknowns” from an $O(N^3)$ generic linear-solve problem into an $O(N)$-sequential-step forward substitution (each step is a small, cheap update, not a full matrix inversion) — exactly the numerical-linear-algebra trick the WY transform is named for.

Compare this with a chain (the $k=1,1,1,\dots$ single-branch case DeltaNet’s original paper handled): there, “ancestor” just means “everything before you,” so $G$ is the ordinary strictly-lower-triangular matrix over token order, and the paper’s tree formulation of Eq. (2) is a strict generalization — a chain is simply a tree with no branching, and DFS pre-order over a chain is just left-to-right token order, so Algorithm 1 degenerates exactly to DeltaNet’s chain WY transform when $N = k$ and there are no branches. This is the sense in which TreeWY is not a new special-purpose trick but a genuine generalization of an existing chain-parallelization method to trees.

3.3 Step 3 — Reconstructing only the accepted state on commit

Once the target model’s verification decides node $a$ is the last accepted node along its root-to-leaf path, the paper reconstructs the continuation state directly from the already-solved $\tilde V$, by restricting the sum in Eq. (5) to $a$‘s ancestor chain:

$$S_a = g_a S_0 + \sum_{i \preceq a} \frac{g_a}{g_i}\, \tilde v_i k_i^\top, \tag{6}$$

where $i \preceq a$ ranges over $a$ itself and all of its ancestors (i.e., the single root-to-$a$ path through the tree, not the whole tree). This becomes the new $S_0$ for the next speculative round. The design choice worth dwelling on here — why store $\tilde V \in \mathbb{R}^{N \times d_v}$ rather than the $N$ full states directly — is where the actual memory win comes from: a full state $S \in \mathbb{R}^{d_v \times d_k}$ has $d_v \times d_k$ entries (16,384 at $d_v=d_k=128$), while a single pseudo-value row $\tilde v_i \in \mathbb{R}^{d_v}$ has only $d_v$ entries (128) — a $d_k = 128\times$ reduction in per-node storage, which is exactly why storing all $N$ nodes’ pseudo-values ($N \times d_v$ total) plus doing one cheap reconstruction on commit ends up so much cheaper than storing $N$ full snapshotted states.

The obvious alternative, and why it doesn’t work as well: one might ask why not just recompute $S_a$ directly by re-running the recurrence of Eq. (3) sequentially along the accepted path, instead of solving the full tree system up front? The answer is that verification happens in one parallel forward pass over the entire tree simultaneously (that’s the whole point of speculative decoding — using otherwise-idle compute) — you don’t yet know which node will be accepted until after that parallel pass computes every node’s output and the target model’s rejection sampling (Eq. 1) runs on all of them. So the per-node outputs $o_t = S_t q_t$ for every node in the tree must already be available before verification’s outcome is known, which is exactly what Eq. (5)/Algorithm 1 provides for all $N$ nodes at once; only the reconstruction step (Eq. 6, needed only for the one node that turns out to be accepted) can be deferred until after the accept/reject decision.

Where this design still costs something (a design tradeoff, not a flaw): building $G$ requires an $N \times N$ ancestor mask and $N$ pairwise dot products $k_t^\top k_i$ for all ancestor pairs — this is more computation than a chain-only verification kernel would need (a chain only ever needs $N$ dot products against the immediately preceding node under naive recurrence, though DeltaNet’s own chain WY transform already pays a similar $O(N^2)$ cost for the same triangular-solve reason). The paper’s own Section 4 discussion of implementation cost (see §4 below) reflects exactly this: a non-causal ancestor mask cannot currently be captured into a CUDA graph the way a simple causal (chain) mask can, so real trees ($w>1$) fall back to a slower, non-graph-captured execution path — the closed-form math generalizes cleanly from chain to tree, but the systems engineering to make a tree as fast as a chain is not yet solved, and the paper is explicit about this being future work rather than claiming a free lunch.

4. Implementation and Experiment Setup

TreeWY is implemented as a fork of vLLM’s main branch (not yet upstreamed at the time of writing), exposed through two SpeculativeConfig options: mamba_state_commit="reconstruct" swaps the default "store_all" snapshotting strategy for the reconstruct-on-commit strategy of §3.3, and draft_tree_widths turns a plain chain into a tree by specifying the branching factor at each depth. A chain (or a “tree” with branching factor 1 everywhere, which is really just a chain in disguise) verifies and commits inside one fused, CUDA-graph-capturable Triton kernel — this is the regime where the paper reports its headline throughput/TTFT numbers. A genuine tree ($w>1$) needs the non-causal ancestor mask described in §3.2, which cannot currently be replayed from a CUDA graph; vLLM falls back to “piecewise” (non-graph) execution for the whole model in that case, which the paper is careful to point out costs far more than the mask computation itself would justify, because it also evicts the GDN mixer kernel from graph capture. This is why the paper explicitly reports tree-width results (Table 2 in the paper, reproduced in §5.3 below) as “enabled and correct, not as a speedup” — an honest and unusually clear-eyed disclosure that the systems engineering has not caught up with the math for the tree case.

Trees also have a scheduling wrinkle: a DFS prefix of a tree is a different topology from the full tree (unlike a chain, where any prefix of a chain is still a valid, shorter chain), so a request whose full tree doesn’t fit that step’s token budget is skipped for speculation entirely that step, rather than being truncated to a smaller tree. All experiments use greedy drafting and verification, and correctness is checked two ways: the closed-form solve matches the literal per-node recurrence to about $10^{-15}$ relative error in fp64 and $10^{-7}$ in fp32, and the production bf16 kernel matches that fp64/fp32 reference within bf16 tolerance — the authors are explicit that this means output token streams are not bit-identical to the snapshotting baseline (floating-point non-associativity across different computation orders), so they gate correctness by comparing acceptance length against a shared no-speculation reference rather than requiring literal bit-identical tokens between the two speculative implementations.

Models and hardware. Two scales of one hybrid model family, Qwen3.5-35B-A3B (30 GDN + 10 softmax layers, tensor-parallel degree 1) and Qwen3.5-397B-A17B (45 GDN + 15 softmax layers, tensor-parallel degree 8), both served on B200 GPUs (178 GiB HBM/device), both with a fixed depth-3 MTP draft chain for the main sweep. Baselines: storeall (vLLM’s current default full-state snapshotting) is the primary baseline; ReplaySSM (deferred-materialization with a rank-1 cache) is compared separately in the paper’s Appendix D on an identical 35B sweep. Workloads: six total — ShareGPT, spec-bench, BurstGPT, and three synthetic profiles (balanced-chat, generation-heavy, summarize-heavy) — swept over gpu_memory_utilization (gmu) $\in \{0.6, 0.75, 0.9\}$ and, for five of the six, max-concurrency $\in \{1,8,32,64,128,256\}$ (BurstGPT instead sweeps Poisson arrival rate). Why this design is a reasonable choice: sweeping both the memory budget (gmu) and the offered load (concurrency) independently is exactly what’s needed to expose a “regime-dependent” effect — the paper’s central empirical claim is that TreeWY’s benefit is conditional on memory being the bottleneck, and you can only demonstrate that convincingly by showing both the memory-bound region (where it wins big) and the non-memory-bound region (where it costs a few percent) rather than reporting a single averaged number that would hide the conditionality entirely.

5. Results & Analysis

5.1 The core mechanism: freed KV/state headroom

Figure 4 above (the paper’s own Fig. 1, reproduced) is the mechanism the rest of the results cascade from: at every concurrency level tested, storeall’s peak KV-cache usage climbs to the pool’s 100% ceiling well before TreeWY’s does, because every one of storeall’s $N+1$ snapshotted states is competing for the same HBM pool as the KV cache and the batch of in-flight requests. Concretely, at the tightest measured budgets (35B at gmu 0.6, 397B at gmu 0.75), TreeWY’s geomean KV-cache reduction over five workloads ranges from about 1.0x (low concurrency, where nobody is memory-bound yet) up to roughly 2.4-5.6x at higher concurrency, where the freed memory actually starts mattering.

5.2 Throughput and latency: a genuinely regime-dependent win, not a fixed multiplier

The paper’s most intellectually honest move is Figure 3 (reproduced conceptually via the paper’s own decomposition numbers below), which splits every one of its ratio metrics by whether the storeall baseline had actually run out of KV headroom at that specific (workload, concurrency, gmu) point, rather than reporting one aggregate number. At 35B, 31 of 105 measured points are memory-bound; there, TreeWY reaches 1.15x throughput, 2.94x lower p99 TTFT, and 1.17x lower mean end-to-end latency, at a cost of 0.83x per-token (TPOT) efficiency — i.e., each individual token comes out a bit slower to produce, but the system admits so many more concurrent requests into the larger effective batch that overall throughput and latency both improve. At 397B, 17 of 70 points are memory-bound, showing the identical qualitative shape at smaller magnitude (1.06x throughput, 1.66x lower TTFT). At the remaining “headroom” points (baseline wasn’t memory-constrained anyway), every metric sits within a few percent of parity by construction — TreeWY isn’t winning anything there because there was nothing to win, but it also isn’t losing anything meaningful.

Why this decomposition matters, and what it would hide if omitted: if the paper reported one averaged number across all 175 (workload, concurrency, gmu) points, the memory-bound wins (up to 30-40x TTFT improvement, Table 1 in the paper) would be diluted by the much larger number of headroom points where nothing changes, producing a headline number that understates the technique’s value in the regime it’s actually designed for, while simultaneously making it look like a bigger unconditional win than it really is. The paper’s own p99 TTFT numbers make the magnitude vivid: at 128 concurrency, 35B, gmu 0.6, TreeWY is roughly 40x faster (683ms vs 27,489ms) — but this number is meaningless without knowing it’s specifically the point where storeall’s pool has saturated and is queuing requests while TreeWY’s hasn’t yet.

5.3 Tree width: affordability, not yet speedup

Table 2 of the paper (Figure 6 above) is arguably the more novel empirical result, because it’s the one that a snapshotting baseline genuinely cannot afford at all past a certain width, rather than merely being more expensive at: growing the tree from a chain (shape (1,1,1), $N=3$) to a fully-branching depth-3 tree (shape (3,3,3), $N=39$) makes acceptance length rise from 3.24 to 3.58 (a real quality gain — more tokens accepted per verification round, because the tree gives the drafter more chances to guess a matching token at each depth), while storeall’s snapshot cost grows 10x (4 to 40 blocks per request per GDN layer) over that same range, whereas TreeWY’s storage cost stays flat at one block regardless of $N$. This is the “affordability” argument, distinct from the “same load, less memory” argument of §5.1: it’s not that TreeWY makes a fixed workload cheaper, it’s that TreeWY makes previously-infeasible configurations (wide trees) feasible at all. The paper is careful to caveat, per §4 above, that this affordability is not yet a throughput win on its own, because a wider tree pushes $N+1$ tokens per verification step through the target model and currently runs on the slower non-graph-captured path — the width sweep demonstrates a capability unlock, not a latency improvement, and conflating the two would overstate the result.

5.4 Comparison against ReplaySSM

Appendix D’s head-to-head against ReplaySSM (the closest concurrent chain-only competitor, which defers state materialization to a periodic flush rather than solving a different triangular system — recall from §2.6/prior-work discussion that inside its verify window it solves the same triangular system TreeWY does) is the paper’s most self-critical section: on the identical 35B sweep, both methods free comparable KV headroom (TreeWY 0.96x ReplaySSM’s peak usage — essentially a tie), but ReplaySSM wins on raw throughput at every concurrency level tested (e.g., 1.12-1.20x its own baseline vs. TreeWY’s 0.99-1.08x). The paper’s own attribution — that the gap is about when state gets materialized (ReplaySSM defers the write; TreeWY commits on every verification step) rather than about the shared triangular-solve math — is explicitly flagged by the authors as “an attribution, not a validated claim,” since they have not implemented a deferred-write commit path themselves to test whether it closes the gap. This is a refreshingly candid admission that TreeWY, as currently implemented, is not simply better than the closest competing approach in every dimension — it wins on affordability and tree support, but currently loses a modest amount of raw chain throughput to a system that defers writes more aggressively.

6. Limitations & Boundary Conditions

What the authors state explicitly. (1) The tree verification path is not yet graph-capturable, so wider trees run on a slower execution path whose cost currently exceeds what the extra acceptance length buys back — tree width is “enabled and correct, not a speedup.” (2) The method has only been validated on one model family (Qwen3.5, at two scales) — the paper explicitly lists “extending to a second model family” as next-step future work, meaning generality across different GDN hyperparameterizations, decay/write-strength ranges, or entirely different linear-attention variants (e.g., non-GDN gated recurrences) is untested. (3) The derivation depends specifically on the gated delta rule’s exact algebraic form (decay-times-rank-1-correction) — it does not automatically extend to, say, a state-space model with a genuinely different (non-rank-1) update structure, though the authors argue the same underlying trick (rewrite as decayed additive attention with corrected pseudo-values) is likely to generalize to other linear-attention recurrences with similar structural properties. (4) All benchmarks use greedy drafting/verification only — sampling-based (non-greedy) speculative decoding, which is common in production for output diversity, is not evaluated, and it’s unclear whether the acceptance-length-matching correctness argument extends cleanly to a stochastic acceptance rule. (5) All evaluations run with prefix caching disabled, which the paper does not explain or justify, despite prefix caching being a common production optimization that could plausibly interact with either baseline’s memory accounting.

What the paper does not fully spell out, but a careful reader should notice. The $O(N^2)$ cost of building the ancestor-gated Gram matrix $G$ (an $N\times N$ mask plus $N$ pairwise dot products for all ancestor pairs) means that as tree width grows, the verification-time compute cost of TreeWY itself grows faster than the naive per-node recurrence would (which is only ever $O(N)$ total dot products against each node’s immediate predecessor). The paper’s width results in §5.3 measure acceptance-length gains and memory-affordability, but do not report the wall-clock cost of the triangular solve itself as tree width scales into the dozens of nodes — a reader is left inferring, rather than being shown, where the $O(N^2)$ construction cost would eventually dominate relative to the piecewise-execution overhead the paper does discuss. Similarly, the ReplaySSM comparison in §5.4 is run only at 35B — it is not clear from the paper whether the throughput gap to ReplaySSM widens, narrows, or stays constant at 397B scale, where TreeWY’s own headline numbers are already noticeably smaller than at 35B (1.06x throughput vs. 1.15x, 1.66x TTFT vs. 2.94x) — a pattern that itself deserves more discussion than the paper gives it, since a technique whose benefit shrinks with model scale is a meaningfully different practical proposition than one whose benefit is scale-invariant.

7. Critical Analysis

(a) Weaknesses and flaws specific to this paper. First, the central mechanism (§3) is entirely orthogonal to what actually produces the paper’s most impressive numbers (up to ~40x TTFT reduction) — those numbers come from admitting more concurrent requests into a batch once memory pressure is relieved, which is a systems/scheduling effect that would arise from any method that frees comparable memory, not something specific to the closed-form triangular solve. The paper is reasonably careful about this (§5.2’s regime decomposition makes it explicit), but a less careful reading of the abstract’s “1.49x throughput, ~30-40x TTFT” headline numbers could easily overstate how much of that gain is attributable to the elegance of the WY-transform derivation itself versus simply “using less memory per speculative attempt,” which any sufficiently memory-frugal alternative (including, per §5.4, ReplaySSM, which achieves comparable memory reduction via an entirely different deferred-write mechanism) would also unlock. Second, the tree-width result — arguably the paper’s most genuinely novel empirical contribution, since the chain-mode memory reduction is conceptually shared with ReplaySSM — is explicitly not a speedup in the current implementation, meaning the paper’s most interesting claim (wide trees become affordable) currently ships with a footnote that the payoff is not yet realized in wall-clock terms; this significantly tempers how much practical value a reader should expect today versus after the graph-capture engineering work the paper defers to future work. Third, the concurrent-work landscape (§2, “existing approaches”) lists Bole as “the closest architecturally” but explicitly states no head-to-head comparison was run because Bole’s paper doesn’t link to code — this leaves the paper’s positioning relative to its most directly comparable competitor entirely unverified, based only on Bole’s own self-reported numbers (82-99x transient-memory reduction, 3.4-7.7x faster tree verification), which is a substantially larger claimed win than TreeWY reports for its own tree case, and the paper offers no explanation for why an ostensibly similar technique would report such different magnitudes.

(b) Limitations the authors understate or omit. The single-author, single-institution (Thomson Reuters) nature of this work, combined with an unmerged, not-yet-upstreamed vLLM fork, means the practical reproducibility bar for an outside practitioner is currently quite high — there is no indication of a public code release, and “implemented as a fork of vLLM’s main branch (not yet upstreamed as of this writing)” is the only implementation detail given about how to actually obtain and run the code. The paper reports GPU-hour totals (≈85 GPU-hours across three sweeps) which is a nice transparency gesture, but doesn’t report how many tuning runs or failed configurations preceded the reported sweeps, which matters for anyone trying to estimate the true engineering cost of adopting this approach versus simply waiting for ReplaySSM (already in a vLLM RFC and TensorRT-LLM) or Bole (already shipped in SGLang) to mature, both of which appear closer to production-ready today than TreeWY’s own unmerged fork. Additionally, while the paper carefully separates “memory-bound” from “headroom” points in its own results, it does not discuss what fraction of real-world production traffic actually falls into the memory-bound regime in practice — the entire value proposition rests on an assumption (memory pressure is common enough to matter) that the paper doesn’t independently substantiate with, say, a real deployment trace’s utilization distribution, as opposed to the concurrency sweeps constructed specifically to produce memory-bound points.

(c) Concrete, specific improvement suggestions. (1) Report wall-clock triangular-solve cost as an explicit function of tree width $N$ (not just acceptance length and block-count), so readers can independently estimate the crossover point beyond which the $O(N^2)$ Gram-matrix construction would offset the acceptance-length gains, even before the graph-capture engineering is finished — this is a purely-measurement addition that doesn’t require new engineering. (2) Run the identical 35B ReplaySSM comparison at 397B, since the paper’s own numbers already hint that TreeWY’s benefit shrinks with scale, and it would be valuable to know whether the same holds for the ReplaySSM comparison specifically (i.e., does ReplaySSM’s throughput lead over TreeWY also shrink, stay constant, or grow at 397B?). (3) Since the deferred-write hypothesis for ReplaySSM’s throughput edge is explicitly flagged as unvalidated, prototyping even a minimal deferred-write variant of the reconstruct-on-commit strategy (write the accepted state to persistent storage only every $m$ verification steps rather than every step) would directly test the authors’ own stated hypothesis and either confirm or refute it, turning a speculative attribution into a validated finding. (4) Given that Bole reports substantially larger memory-reduction and speedup numbers on a nominally similar technique, even a partial, non-head-to-head sanity check — e.g., re-deriving Bole’s reported memory-reduction formula from its paper’s stated method and checking whether it is measuring the same thing TreeWY measures (peak-usage reduction under otherwise-identical serving conditions, vs. some more favorable metric definition) — would materially strengthen the paper’s positioning claims rather than leaving the discrepancy unaddressed.

8. Reproducibility & Practical Notes

There is no public code release as of this review; the implementation is described only as “a fork of vLLM’s main branch (not yet upstreamed as of this writing),” so a practitioner cannot currently run TreeWY without either re-implementing the two-step algorithm (build the ancestor-gated Gram matrix, forward-substitute, reconstruct on commit) from Eq. (2)-(3) of the paper (Eq. 4-6 of this review) directly against a specific inference engine’s GDN kernel, or waiting for the authors to upstream their fork. The derivation itself, however, is engine-agnostic and depends only on the gated delta rule’s algebraic form — any serving stack that already has a GDN/gated-linear-attention layer (vLLM, SGLang, or a custom stack) could in principle implement the same triangular-solve trick, since the paper is explicit that “the derivation depends only on the gated delta rule, not on any other architectural detail.” For someone wanting to prototype this today, the most tractable starting point is probably the chain-only case (no tree), since it maps directly onto the already-published, already-implemented DeltaNet chain WY transform (Yang et al., arXiv:2406.06484) — extending an existing chain WY implementation to add the tree-ancestor mask of Eq. (2) is a considerably smaller lift than building the whole system from scratch, and sidesteps the graph-capture engineering problem entirely (since chains are already graph-capturable per §4). Compute requirements to reproduce the reported results are non-trivial: the paper’s own accounting is ≈85 GPU-hours on B200 GPUs (178 GiB HBM/device) across the three reported sweeps, excluding one-time model-loading overhead, and requires access to both scales of the Qwen3.5 model family (35B-A3B fits on a single B200; 397B-A17B needs TP8, i.e. 8 B200 GPUs).

8b. Related-Work Positioning in One Table

To make the four alternative approaches the paper discusses (§2 of the paper) easy to compare at a glance, here is a consolidated summary:

Approach	Handles trees?	Handles GDN’s non-commuting transition?	Where state is materialized	Code available?
Full-state snapshotting (vLLM/SGLang default)	Yes	Yes (trivially, no math needed)	Every draft position	Yes (shipping default)
ReplaySSM (Dao AI Lab / NVIDIA)	No (chain-only implementation)	Yes (covers Mamba2 and GDN)	Deferred, periodic flush	Yes (vLLM RFC, TensorRT-LLM PR)
STree	Yes	No (Mamba2 scalar-decay only)	Recompute via cumulative sum	Partially (research code)
Bole (SGLang)	Yes	Yes (closed form, general hybrid-attention)	Tree-structured closed form	No (paper only, no linked code)
TreeWY (this paper)	Yes	Yes (tree-structured WY/UT transform)	Reconstruct only accepted node on commit	No (unmerged vLLM fork)

Reading this table against §5.4’s ReplaySSM comparison and §2’s Bole discussion, TreeWY is the only entry that is simultaneously tree-capable, GDN-native, and grounded in a formally derived closed form — but it is also the only entry among the “yes/yes” row and Bole without any linked, runnable implementation, which is precisely the reproducibility gap flagged in §6 and §7(b).

9. Conclusion

TreeWY’s core contribution is a clean piece of algebra: recognizing that the gated delta rule is, after a re-grouping of terms, just decayed additive attention over a set of correctly-computed pseudo-values, and that this fact — already known and exploited for training-time parallelization by plain DeltaNet — also solves the speculative-verification rollback problem for GDN layers, provided the “chain” generalization is worked out for a “tree.” The practical payoff is real but conditional: where a serving system’s KV/state memory pool is genuinely the bottleneck, TreeWY turns freed HBM into meaningfully higher throughput and dramatically lower tail latency (up to ~30-40x p99 TTFT in the paper’s most memory-starved configurations); where memory isn’t binding, the technique costs a small, honestly-reported throughput tax. The tree-width result is the more novel contribution scientifically (a wider, higher-acceptance draft becomes storable at all, not just cheaper), but its systems-engineering payoff — a graph-capturable tree-verification kernel — remains future work rather than a delivered result today. Combined with a candid, unresolved comparison against a closely competing concurrent approach (ReplaySSM) that currently wins on raw chain throughput via a different mechanism, this is a paper whose mathematical contribution is more mature and more clearly established than its systems-engineering and competitive-positioning story, which is honestly presented as still in progress.

References

1. Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML, 2023.
2. S. Yang, J. Kautz, A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025.
3. S. Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484, 2024.
4. Y. Wu et al. STree: Speculative Tree Decoding for Hybrid State-Space Models. arXiv:2505.14969, 2025.
5. L. Wang et al. Bole: Efficient Tree Speculation for Hybrid-Attention Language Models. arXiv:2608.01651, 2026.
6. Dao AI Lab and NVIDIA. ReplaySSM: Cache SSM Inputs, Not State. Blog / vLLM RFC #47572, 2026.
7. W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). SOSP, 2023.