FreeToken: Turning Your Gaming PC Into a Frontier-Model Serving Box

Review date: 2026-08-23 Author: Zhongzhu Zhou Paper reviewed: FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution Paper authors: Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica (UC Berkeley, MIT, UT Austin, and collaborators) arXiv: 2608.16157 Venue/Status: Preprint (cs.DC), August 2026

1. The idea in one sentence, and why it’s not a small deal

FreeToken is a serving engine that lets a single personal machine — a laptop, a gaming desktop, or a workstation with one GPU — run frontier-scale open-weight Mixture-of-Experts (MoE) models that are far larger than the machine’s VRAM, at genuinely interactive speed, by treating the GPU, the CPU, host DRAM, and the PCIe interconnect not as separate resources to be individually optimized but as one unified, continuously re-balanced inference platform. The headline numbers are almost comic in their scale mismatch: a 753-billion-parameter model (GLM-5.2) served on a single workstation GPU, a 284B model served interactively on a 32GB gaming desktop, and a 35B model served faster than production Codex’s median decode speed on an 8GB laptop GPU.

What makes this a genuine systems contribution rather than a “throw more offloading at it” result is a specific insight buried in the decode path: when an MoE layer’s routed experts miss the GPU cache, there are two ways to serve that miss — transfer the expert weights over PCIe and run them on the GPU, or execute the expert directly on the CPU where it already resides in host memory — and neither is universally better, because both draw from the same host-memory bandwidth pool. FreeToken derives a closed-form optimal split (a single scalar, $q^\star$) between these two paths from two numbers measured on the actual deployed machine: PCIe transfer bandwidth and CPU-side expert-execution bandwidth. This turns what looks like a hardware-specific tuning problem into a two-line formula that adapts automatically to whichever consumer GPU, whichever DRAM generation, and whichever PCIe generation happens to be sitting on the user’s desk.

This review is aimed at a reader who understands the basics of transformer inference and KV caching but hasn’t necessarily thought carefully about edge MoE serving specifically — where the constraint isn’t “how do I make one GPU faster” but “how do I make a GPU, a CPU, and a slow interconnect cooperate as if they were one bigger, weirder GPU.” Section 2 builds the prerequisites (MoE sparsity, expert offloading, why prefill and decode are different problems). Section 3 unpacks FreeToken’s three-part design end to end, including the full derivation of the $q^\star$ policy and the semantic-aware caching mechanisms. Section 4 covers the CUDA-graph implementation detail that makes the dynamic policy actually fast in practice. Section 5 walks through every experiment in the paper, not just the headline throughput numbers. Section 6 covers limitations, and Section 7 is a dedicated critical-analysis section.

2. Prerequisites

2.1 Why MoE is both the opportunity and the problem for edge serving

A dense transformer of $N$ parameters requires roughly $N$ parameters’ worth of compute for every token it processes — there’s no way to serve a 700B-parameter dense model faster than a 700B-parameter’s worth of matmuls per token, on any hardware. Mixture-of-Experts (MoE) architectures break this coupling: an MoE layer contains $E$ expert sub-networks (often hundreds), but a lightweight router selects only $k \ll E$ of them for each token. DeepSeek-V4-Flash, for instance, activates 6 of 256 routed experts per layer across 43 layers, so only 13B of its 284B total parameters participate in computing any single token’s output. This is the “opportunity” half of the paper’s framing: sparse activation means the compute required per token is small enough that consumer-class silicon could plausibly keep up, even though the parameter count is enormous.

The “problem” half is that sparsity reduces per-token computation without proportionally reducing the memory required to store the full expert pool — you still need all 256 experts sitting somewhere in memory, because you don’t know in advance which 6 a given token will route to. If the complete pool doesn’t fit in GPU VRAM (and for a 284B or 753B model on a 24–96GB consumer/workstation GPU, it emphatically does not), the inactive experts must live in host DRAM or on disk and get pulled onto the GPU on demand. This is expert offloading, and it is the central systems problem edge MoE serving has to solve: sparse activation makes the computation feasible, but the full expert pool makes efficient serving hard, because now every token’s forward pass potentially requires a round-trip over a comparatively slow PCIe link to fetch weights that aren’t already resident on the GPU.

2.2 Prefill and decode are different bottlenecks, not the same problem at different scales

It’s tempting to think of “serving an MoE model efficiently” as one problem, but the paper’s diagnostic framing (Section 2 of the original) splits it cleanly into two regimes with genuinely different failure modes:

• Prefill (processing the initial prompt, which may be thousands of tokens) activates nearly the entire expert set of every layer, even though any single token only routes to $k$ experts — because with enough tokens in the prompt, the union of their individual routing decisions covers most of the expert pool. A prefill pass therefore has to stream close to the complete expert pool through the CPU–GPU interconnect at least once, which is a pure bandwidth problem: for an FP4 DeepSeek-V4-Flash deployment, that’s roughly 140GB of expert weights, taking about 2 seconds on a fast PCIe 5.0 link and 10+ seconds on the narrower ×8 links common in laptops. An engine that fetches experts purely on-demand exposes this entire multi-second window as GPU idle time — unacceptable for an interactive agent that’s supposed to start responding quickly.
• Decode (generating one token at a time after the prompt) is the opposite regime: each step activates only a genuinely sparse handful of experts, but which experts get selected changes with every token, so a placement chosen once (at load time or after prefill) quickly goes stale, and每次 cache miss forces either a fresh PCIe transfer or CPU execution. The problem here isn’t total bytes moved — it’s serving many small, unpredictable misses with low latency, over and over, forever, using a CPU whose DRAM bandwidth is a small fraction of the GPU’s on-package memory bandwidth (roughly 50–90 GB/s for consumer dual-channel DDR4/DDR5 versus 1–1.8 TB/s for an RTX 4090/5090’s VRAM).

Both problems are further compounded by a third factor that has no datacenter analogue: edge hardware is not dedicated. A user’s browser, games, and other applications compete for the same VRAM and can claim gigabytes of it at any moment; the “budget” available to the serving engine is not a launch-time constant but a moving target throughout the session. Any edge-serving design that assumes a fixed memory budget, chosen once at startup, is solving the wrong problem.

2.3 Existing engines each solve a fragment, not the whole thing

llama.cpp assigns MoE tensors to devices statically at load time. KTransformers pins a fixed “hot” subset of experts on the GPU and executes the rest on CPU with AMX-optimized kernels, updated only at prefill boundaries. MoE-Infinity traces request-level activation patterns to guide prefetching. None of these systems dynamically re-derives, at every decode step, how to divide residual cache misses between “transfer to GPU” and “execute in place on CPU” based on the actual measured bandwidth balance of the deployed machine — and this is precisely the gap FreeToken’s core algorithmic contribution (Section 3.2 below) fills.

3. FreeToken’s design, unpacked

FreeToken organizes the GPU’s memory around a two-level hierarchy (Figure 1 below reproduces the paper’s own architecture diagram): the CPU-resident expert pool holds the complete set of routed-expert weights and is always the source of truth, while non-expert weights (attention projections, layer norms, embeddings) live permanently on the GPU. The remaining GPU memory becomes a single elastic expert cache shared across all MoE layers, where each cache slot holds every tensor needed to evaluate one (layer, expert) pair — meaning residency and lookup operate on logical identifiers, not on tensor shards.

The design maps directly onto the two bottlenecks Section 2.2 identified: prefill needs its transfer hidden behind computation and its recomputation avoided across context edits (Section 3.1); decode needs its residual cache misses divided intelligently between two execution paths (Section 3.2); and both phases need the underlying GPU memory budget itself to be adjustable at runtime rather than fixed at launch (Section 3.3).

3.1 Prefill: full-layer double buffering + semantic-aware state checkpoints

Full-layer double buffering hides transfer behind computation. Because prefill activates nearly every expert in every layer (Section 2.2), on-demand fetching is the wrong strategy — you already know you’ll need almost the whole layer’s expert set, so why wait to find out which fraction misses? FreeToken instead allocates two full-layer buffers from the shared slot pool. While the GPU computes layer $l$‘s routed experts from one buffer, a dedicated transfer stream simultaneously streams the complete expert set of layer $l+1$ into the other buffer — the transfer for layer $l+1$ can start before layer $l+1$‘s actual routing decisions are even known, because you’re fetching (essentially) everything anyway. The two buffers then swap roles for the next layer. This converts what would otherwise be a serial “wait for transfer, then compute, then wait for the next transfer” pattern into a fully pipelined one where PCIe bandwidth utilization approaches its physical ceiling. When the shared slot pool can’t spare two full layers’ worth of space (e.g., a very memory-constrained laptop), the engine gracefully falls back to on-demand prefill loading rather than oversubscribing GPU memory and risking an OOM.

Semantic-aware state checkpoints let recurrent state survive context edits, without re-deriving from scratch. Many frontier models mix full attention with linear/recurrent layers (gated DeltaNet in Qwen3.6, Kimi Delta Attention in Kimi-K3) precisely because full attention’s KV cache grows linearly with context length. These recurrent layers compress the entire past context into a single evolving state, which is efficient for memory but creates a subtler problem: unlike a KV cache (whose prefix can be partially reused if only the suffix of the context changed), a recurrent layer’s single compressed state either matches the current context exactly or it’s useless — there’s no “reuse the first half” option once the state has folded everything together. Because storing a checkpoint of this full state costs as much as hundreds of tokens’ worth of ordinary KV cache, only a small number of checkpoints can be kept, so where you place them matters enormously.

Agentic harnesses modify context at very specific, predictable points: OpenClaw strips old thinking blocks from every turn but the latest; OpenCode truncates old tool outputs beyond a recent window; SWE-agent elides all but the last $n$ observations. In every case, the edit removes or replaces a whole block delimited by a special token boundary (</think>, </tool_call>, </tool_output>), and the prefix up to that boundary survives unchanged. FreeToken exploits this by anchoring its scarce recurrent-state checkpoints specifically at these semantic boundaries, rather than at arbitrary token positions. When a new request arrives after a context edit, it restores from the deepest surviving checkpoint and only needs to re-prefill the genuinely new suffix — full-attention layers reuse their radix-tree-indexed KV cache up to the edit point exactly as an ordinary serving system would, while recurrent layers resume from the anchored checkpoint instead of recomputing their entire compressed state from the beginning of the conversation. Checkpoint slots are LRU-evicted independently of the ordinary KV pool.

3.2 Decode: the $q^\star$ bandwidth-adaptive execution policy, derived step by step

This is the mechanistic heart of the paper, so it’s worth deriving carefully rather than just quoting the final formula.

Setup. At each MoE layer during decode, a GPU kernel identifies which of the token’s routed experts are already resident in the shared LRU cache (the hits, set $H$) versus which are not (the misses, set $M$, with $|M| = m$). Hits execute directly on the GPU — no decision needed there. The interesting question is entirely about how to serve the $m$ misses.

Why not just always transfer misses to the GPU? Because a missed expert doesn’t have to go through PCIe at all — it can instead be executed directly on the CPU, in place, since it’s already resident in host DRAM. The naive approach (used by KTransformers-style static CPU offloading) always executes CPU-resident experts on the CPU; the naive alternative (used by a pure prefetch-and-transfer design) always brings a miss onto the GPU. FreeToken’s insight is that both extremes waste a resource: pure CPU execution leaves the PCIe link and the GPU compute idle even when PCIe has spare bandwidth to spend on a transfer; pure transfer-and-execute leaves CPU cores idle and forgoes the future cache hits that a transfer would eventually buy, but it also competes for the same host-memory bandwidth that CPU execution needs, since both a DMA transfer and a CPU compute kernel ultimately have to read the expert’s weight bytes out of host DRAM.

The residual-bandwidth argument. Let $B_P$ be the measured PCIe expert-transfer bandwidth and $B_H$ be the measured host-side (CPU) expert-execution bandwidth — both profiled empirically on the actual deployed machine, not read off a spec sheet, because as the paper notes, “this optimal mixture cannot be read from specification sheets.” Since both a PCIe transfer and CPU execution ultimately draw on the same host-memory subsystem, once a PCIe transfer is saturated it leaves a residual bandwidth:

$$B_R = \max(B_H - B_P,\ 0). \tag{1}$$

This residual is exactly what’s left over for the CPU to spend on direct expert execution concurrently with the ongoing PCIe transfer. FreeToken splits the $m$ misses into a cache-fill set $F$ (transferred to the GPU, executed there, and retained for future reuse) and a CPU-execution set $C$ (executed in place, leaving GPU residency unchanged), with $M = F \,\dot\cup\, C$ and $q = |F|$. If $S$ is the size in bytes of one expert, the two branches’ execution times are

$$T_{\text{fill}}(q) \approx \frac{qS}{B_P}, \qquad T_{\text{cpu}}(m-q) \approx \frac{(m-q)S}{B_H - B_P}. \tag{2}$$

The two branches execute concurrently — the CPU branch doesn’t wait for the GPU branch to finish — so the exposed latency of the miss-handling step is the slower of the two, $\max(T_{\text{fill}}, T_{\text{cpu}})$. This is minimized (subject to $q + (m-q) = m$ fixed) exactly when the two branches take equal time, i.e. when the system is perfectly load-balanced between them. Setting $T_{\text{fill}}(q) = T_{\text{cpu}}(m-q)$ and solving:

$$\frac{q}{m-q} \approx \frac{B_P}{B_H - B_P} \quad\Longrightarrow\quad q^\star \approx m \cdot \frac{B_P}{B_H}. \tag{3}$$

Sanity-checking the formula at its edges. As $B_H \to B_P$ (host bandwidth barely exceeds transfer bandwidth — a “starved” host), $q^\star \to m$: every miss should go to the GPU, because the CPU has essentially no spare bandwidth to contribute beyond what’s already saturating the PCIe link, so the formula automatically degenerates to pure cache-fill without needing a separate special case. As $B_H \gg B_P$ (a very fast host relative to a slow PCIe link — the 8GB laptop’s PCIe ×8 case), $q^\star$ shrinks and more of the miss-handling work shifts to the CPU, which makes intuitive sense: if PCIe is your bottleneck resource, don’t spend it on every miss when the CPU has bandwidth to spare.

In practice, $q^\star$ is rounded to an integer, at least one fill is always retained (so the cache keeps warming even under heavy CPU load), and the specific experts assigned to $F$ versus $C$ are delegated to the LRU cache’s normal victim-selection logic rather than chosen by the bandwidth formula itself — the formula only decides how many, not which.

A worked numerical example, using the paper’s own measured hardware bandwidths. It’s worth plugging real numbers into Eq. 3 to see how differently $q^\star$ behaves across the paper’s six test systems (Table 1 bandwidths), since this is exactly what explains some of the cross-hardware results in Section 5.4:

System	$B_P$ (GB/s)	$B_H$ (GB/s)	$q^\star/m$	What this means in practice
RTX 4060 laptop	11.8	47.5	0.25	Only ~1 in 4 misses goes to the GPU; the CPU, with 4× the laptop’s narrow PCIe ×8 bandwidth, absorbs most of the remainder.
RTX 3090 (server-capped)	25.3	56.7	0.45	Roughly an even split, slightly CPU-favoring.
RTX 4090 (server-capped)	25.1	63.2	0.40	Similar to the 3090 — PCIe 4.0’s ceiling is well below this system’s host bandwidth.
RTX 5090 (server-capped)	52.7	77.3	0.68	PCIe 5.0 narrows the gap; most misses now go to the GPU.
RTX 5090 desktop	49.0	53.8	0.91	Host and link bandwidth are nearly matched — almost every miss goes to the GPU, because the CPU has almost no residual bandwidth left over once the (nearly-as-fast) PCIe link is accounted for.
RTX PRO 6000 workstation	51.5	178.0	0.29	A huge host-bandwidth advantage (8-channel DDR5) pulls the split back toward CPU execution, despite this being the fastest PCIe link tested.

The pattern that falls out is not “faster GPU link → more GPU offload” in isolation — it’s the ratio of the two bandwidths that matters, which is exactly why a spec-sheet-based heuristic (e.g., “always offload X% to CPU”) would get every one of these six systems wrong in a different direction. It also directly explains the cross-hardware finding from Section 5.4: the RTX 5090 desktop’s weaker (dual-channel, consumer-grade) host memory doesn’t hurt FreeToken much, precisely because $q^\star/m = 0.91$ on that system already routes nearly everything to the GPU path, so a weak CPU-execution path barely gets exercised in the first place — whereas llama.cpp’s static CPU-heavy placement has no such adaptive escape hatch and pays the full cost of that weak host bandwidth.

Execution order and exact correctness. FreeToken launches the CPU branch first (since it has to read weights out of DRAM, giving it a head start relative to the PCIe transfer, which shares that same DRAM read path), then runs the GPU miss path (cache update, batched copy of $F$, grouped evaluation of the combined GPU execution set $G = H \cup F$), while the CPU workers concurrently process $C$. The CPU and GPU partial outputs are merged exactly — there is no algorithmic approximation, no dropped experts, and no precision loss; the model’s output is bit-identical to what a VRAM-resident deployment would have computed. This exactness is worth emphasizing because several competing approaches in the related-work landscape (Section 6 below) trade exactness for speed (fetching lower-precision expert replicas, skipping low-scoring experts); FreeToken’s speedup comes entirely from how the (unmodified) computation is scheduled across CPU and GPU, not from computing something slightly different.

3.3 Semantic-aware expert caching: what makes the LRU cache worth having at all

The $q^\star$ policy only matters for the misses that do occur; the size of $m$ itself is determined by how good the GPU-resident cache is at anticipating which experts the router will pick next. FreeToken observes that decode-time routing exhibits strong temporal locality — across consecutive decoding steps, the same MoE layer tends to route to overlapping or recently-used experts (a phenomenon independently measured across multiple model families in prior “routing consistency” work). This licenses a straightforward but effective policy: a shared LRU cache across all layers, where a hit refreshes an expert’s recency, a fill admits the newly-selected expert, and eviction removes whichever expert has gone longest unused. No workload-specific prediction model, no learned prefetcher — just the classical temporal-locality assumption, applied at the (layer, expert) granularity rather than the (layer, tensor-shard) granularity that a naive implementation might use.

The paper’s later ablation (Section 5.3 below) directly measures how much this buys over the two competing placement philosophies (static split, prefill-time update) — and the gap is large enough (2–5× lower miss rates at matched cache size) that this simple mechanism is doing real work, not just marginal cleanup.

3.4 Elastic memory management: making GPU budget a runtime variable, not a launch-time constant

Because host-resident experts remain the permanent source of truth, GPU cache capacity is purely a performance lever, never a correctness one — shrinking the cache can only make things slower, never wrong. This property is what lets FreeToken do something most serving engines can’t: resize the GPU expert cache at runtime, at scheduler safe points, without restarting the engine or reloading the host-resident pool from disk. This matters concretely because two things drift during a real session: the VRAM budget available to the engine (shared with browsers, games, compositor windows) and the split of that budget between KV cache and expert cache (agentic sessions accumulate context over many turns, so KV-cache demand grows across the session even though the expert working set stays roughly fixed — a split that was correct on turn 1 is wrong by turn 20).

The paper also addresses a second, less glamorous but very real edge-deployment cost: startup latency. Loading a ~140GB expert pool from a 7GB/s NVMe drive takes on the order of 20 seconds before any request can even begin — and on a personal machine, engine startup happens often (users close the engine to free the machine, switch models, restart after a crash), unlike a datacenter deployment that starts once and runs for weeks. FreeToken shortens this by reading expert weights directly from disk into their final host memory layout and pinning the memory only after it’s filled — pinning empty buffers first would fault in and zero out gigabytes of pages that are about to be immediately overwritten, which is pure wasted work. It further eliminates GPU warmup entirely: the very first request is served with a cold cache, its misses handled by the ordinary decode path described in Section 3.2, and the cache heats up naturally through normal serving rather than through a dedicated (and also user-visible) warmup phase.

3.5 Putting the decode-step control flow together

The prose walkthrough above is easiest to internalize as a single diagram of what happens, in order, on every MoE layer of every decode step:

flowchart TD
    A["Router selects top-k experts for current token"] --> B["GPU kernel classifies each against the residency table"]
    B --> C{"Resident in GPU cache?"}
    C -- "Hit (set H)" --> D["Execute directly on GPU, refresh LRU recency"]
    C -- "Miss (set M, |M|=m)" --> E["Compute q* = m * B_P / B_H (Eq. 3)"]
    E --> F["Split M into cache-fill set F (size q) and CPU-exec set C (size m-q)"]
    F --> G["CPU branch: execute C in place from host-resident pool"]
    F --> H["GPU branch: transfer F over PCIe, update cache, evaluate G = H_hits union F"]
    G --> I["Merge CPU and GPU partial outputs exactly (no approximation)"]
    H --> I
    D --> I
    I --> J["Layer output, bit-identical to a fully VRAM-resident deployment"]

Figure A (self-drawn, summarizing Section 3.2’s decode control flow): every layer’s miss set is divided by the closed-form $q^\star$ ratio into a GPU cache-fill path and a CPU in-place-execution path, which run concurrently and merge exactly. This is the mechanism that turns a hardware-specific tuning question into a two-measurement, closed-form runtime decision, re-evaluated fresh at every layer of every step.

3.6 How the elastic-memory and fast-bootstrap mechanisms fit together

Section 3.4’s two mechanisms — runtime cache reconfiguration and fast bootstrap — are easiest to see as two phases of the same lifecycle, one at startup and one continuously during serving:

flowchart TD
    subgraph startup["Engine bootstrap (Section 3.4)"]
        A["Read expert weights from disk directly into final host bank layout (FTW format)"] --> B["Pin host memory only after banks are filled"]
        B --> C["Serve first request with a cold GPU cache — no dedicated warmup phase"]
        C --> D["Cache heats up through the ordinary decode path (Section 3.2)"]
    end
    subgraph runtime["Continuous runtime adaptation, at scheduler safe points"]
        E["VRAM budget shifts: other apps claim/release GPU memory; KV-cache demand grows across agent turns"] --> F["Rebuild GPU expert cache for the revised budget"]
        F --> G["No engine restart; no reload of the host-resident expert pool"]
        G --> E
    end
    D --> E

Figure B (self-drawn, summarizing Section 3.4): fast bootstrap gets the engine serving quickly without a dedicated warmup, and the same “host pool is always correct, GPU cache is only a performance lever” property lets the cache be resized indefinitely at runtime without ever restarting the engine. Both mechanisms rest on the same underlying invariant — GPU-side state can never be a source of correctness, only of speed — which is what makes both of them safe to do without pausing the serving loop.

4. Implementation: making a routing-dependent policy fast enough to matter

A clever bandwidth-split formula is worthless if computing and applying it re-introduces the very latency it’s trying to eliminate. FreeToken’s implementation section addresses exactly this risk, and it’s worth understanding why it’s a nontrivial problem before appreciating the fix.

The problem: routing-dependent control naively requires host synchronization every layer. Which experts miss, how many miss, and which cache slots get evicted all change at every single MoE layer, every single decode step — this is inherently dynamic, data-dependent control flow. A naive implementation would need the GPU to finish computing routing decisions, copy that result back to the host CPU, have host code decide $q^\star$ and select victims, and then issue the next kernel launches — a costly device synchronization on every layer, which would eat most of the latency savings the whole scheme was designed to capture.

The fix: keep all routing-dependent control on the GPU, as data inside a statically captured CUDA Graph. For each MoE layer, a single GPU kernel deduplicates the routed experts, classifies them against the residency table, computes the bandwidth-derived fetch count $q$, selects eviction victims, and rewrites logical expert IDs into physical slot IDs (or a CPU-assignment flag) — all without leaving the device. Victim selection specifically avoids the classic LRU pitfall of needing a full cache scan per evicted slot: a single-pass kernel identifies the $K$ least-recently-used candidate slots in one shot, and the miss path simply consumes the first $q \le K$ of them, so victim-discovery cost is constant regardless of how many misses actually occur on a given step. Because every expert bank shares the same logical (layer, expert)-to-slot mapping, one device-resident index list drives a single fused transfer across all banks in one fixed-shape kernel launch, with a valid-count field masking unused work — this yields few kernel launches, high sustained PCIe utilization, and zero host-side decision latency in the hot path.

The CPU branch is captured into the same graph: for each supported decode batch size, FreeToken prepares stable pinned I/O buffers and persistent task descriptors, so the device-to-host copy, a host-function submit node, the concurrent GPU path, a synchronization node, and the host-to-device result copy are all captured together and replayed as one unit — meaning replay re-executes the entire heterogeneous step (GPU and CPU branches together) without any per-token Python-level scheduling overhead. The CPU-side worker pool is a persistent, physically-core-pinned C++ pool whose kernels use architecture-specific SIMD and in-kernel dequantization, keeping the whole path bandwidth-bound rather than compute-bound (which matters because the whole premise of the $q^\star$ formula assumes CPU execution is limited by memory bandwidth, not by CPU compute throughput).

On the storage side, FreeToken introduces the FTW (FreeToken Weight) format, which pre-merges model-specific checkpoint layouts into a small set of “expert banks” indexed by a flattened $(layer, expert)$ identifier, so both GPU kernels and the CPU executor address experts through one shared logical identity regardless of the original checkpoint’s physical tensor layout. This lets engine startup skip tensor discovery and repacking entirely, reading aligned chunks straight into exact-size host banks via parallel direct I/O. When the full expert pool can’t be pinned or registered for DMA (a restriction on some OS/driver configurations), the engine falls back to a pure-CPU MoE backend where expert weights stay in ordinary pageable memory and all routed experts execute on CPU — trading peak bandwidth for guaranteed deployability, an explicit and reasonable degradation path rather than a hard failure.

5. Experiments, unpacked

5.1 Setup

Six discrete-GPU systems span the practical range of consumer and workstation hardware (Table 1 in the original paper): an RTX 4060 laptop (8GB VRAM, PCIe ×8, 11.8 GB/s measured transfer bandwidth), an RTX 5090 desktop and three rented dual-socket servers (3090/4090/5090, capped to 6 CPU threads and pinned to the GPU’s NUMA node to emulate consumer-scale host bandwidth despite server-class CPUs), and a single RTX PRO 6000 Blackwell workstation GPU (96GB VRAM) for the frontier-scale demonstration. All bandwidth numbers in the paper’s tables are measured on the deployed tensor shapes, not taken from vendor spec sheets — a methodologically important detail given that the whole $q^\star$ formula depends on accurate bandwidth measurement.

Two primary models: DeepSeek-V4-Flash (284B total, 13B active, natively MXFP4-quantized routed experts) and Qwen3.6-35B-A3B (BF16, except the 8GB laptop which uses its official NVFP4 release for memory reasons). A cross-hardware extension adds GLM-5.2 (753B total, 40B active, NVFP4, a 433GB checkpoint) on the RTX PRO 6000 as the frontier-scale tier. Four realistic agentic workloads probe different serving regimes: W1 (AIME math reasoning, single-turn, decode-dominated, no tool use), W2 (a SWE-bench coding issue solved via the OpenCode harness with real tool execution across three scripted turns), W3 (the same coding issue via Claude Code’s native protocol, which spawns concurrent subagents and grows sessions to 56–65k tokens), and W4 (a 13-turn email/calendar agent via OpenClaw, carrying a ~24.5k-token system-context floor). Baselines are llama.cpp, Ollama, KTransformers, and MoE-Infinity, each on the configurations they support, with weight formats aligned exactly across engines for fairness.

5.2 Headline result: decode throughput and the tail-latency story

On the RTX 5090, FreeToken sustains 77–83 tok/s on Qwen3.6-35B-A3B and 22–25 tok/s on DeepSeek-V4-Flash — 1.8–2.3× and 1.5–1.9× the strongest baseline in each workload, respectively. What’s more striking than the raw numbers is stability under agentic load: FreeToken’s decode rate stays within 12% of its single-turn W1 value across the three genuinely agentic workloads (W2–W4), while the most context-sensitive baseline, KTransformers on DSV4-Flash, has already lost 31% of its W1 rate by W2 alone. This matters because single-stream micro-benchmarks systematically overstate real-world baseline performance — a serving engine that looks competitive on an isolated single-turn benchmark can degrade sharply the moment context grows and tool calls start triggering re-prefills, which is exactly the regime real coding and email agents live in. MoE-Infinity, notably, only manages to serve W1 at all (8.8 tok/s): its per-expert prefill staging cap aborts on the longer-prompt workloads, and it retains no KV cache across requests in its bundled server, making it a poor fit for anything multi-turn.

Time-to-first-token tells an even sharper story about availability, not just speed. FreeToken posts the lowest mean TTFT in five of the six multi-turn cells (only losing to KTransformers’ dedicated GPU-prefill arm on Qwen3.6×W3, and to llama.cpp on W1’s short, isolated prompts where there’s little to hide behind pipelining anyway). The tails separate the engines far more dramatically than the means: FreeToken’s worst-case turn stays below 44 seconds in every single cell tested, while every baseline exceeds 150 seconds somewhere — llama.cpp hits 232s, Ollama 179s, and KTransformers a startling 946s. These aren’t just “slow” numbers; they cross real thresholds where actual clients give up: OpenClaw ships a 120-second idle watchdog, and Claude Code’s default request timeout is roughly ten minutes. A baseline that occasionally exceeds these thresholds isn’t merely slower — it silently fails the request from the user’s perspective. This is the paper’s strongest argument for treating tail TTFT as an availability boundary, not a latency statistic to be averaged away.

5.3 Attributing the gains: pipelined prefill and expert-cache locality

Pipelined prefill. With the double-buffer overlap enabled, each 8,192-token prefill chunk completes in 1.19–1.22 seconds — which is exactly the time needed to stream the model’s full 64.4GB expert pool once at the measured PCIe ceiling of 52.7 GB/s. In other words, expert computation is fully hidden behind the transfer; prefill becomes purely transfer-bound, and throughput climbs to 6.7k tok/s at a 16k-token prompt. Disabling the second buffer (serializing transfer against compute) costs 19% of throughput at 4k tokens, 25% at 8k, and 26% at 16k — the penalty grows with prompt length, because longer prompts have proportionally more computation to hide behind transfer, and losing that overlap wastes a growing fraction of it.

Expert-cache locality. Replaying identical decode-time routing traces from all four workloads against the three engines’ placement policies, at the RTX 5090’s actual serving cache capacity (37% of Qwen3.6’s expert pool, 11% of DSV4-Flash’s — a meaningfully undersized cache relative to the full model, which is exactly the realistic edge regime), FreeToken’s global LRU misses only 16% and 39% of decode-time expert reads on the two models respectively, versus 41%/59% for KTransformers’ prefill-updated placement and a steep 62%/89% for llama.cpp’s routing-blind static split. This ordering — LRU beats periodic-update beats fully-static — holds across every workload at every cache capacity short of holding the entire pool, which is a fairly clean confirmation that temporal locality in MoE routing is real and exploitable, not an artifact of one specific workload or model.

5.4 Cross-hardware generalization and the frontier-scale demonstration

The advantage generalizes across the full range of consumer hardware, not just the RTX 5090 used for the main breakdown: FreeToken leads the strongest available baseline by 1.3× on both the RTX 3090 and RTX 4090, 1.9× on the (server-emulated) RTX 5090, 2.1× on the real RTX 5090 desktop, and 1.8× on the RTX 4060 laptop — where the NVFP4 build sustains 39.3 tok/s on an 8GB, PCIe ×8 machine, which is 92% of the RTX 4090’s rate despite dramatically weaker hardware, and notably exceeds the 33 tok/s median decode speed measured for production Codex traffic. A particularly clean natural experiment sits in the two “5090” columns: the server-pinned RTX 5090 and the real RTX 5090 desktop share identical GPU silicon and differ only in the host system. Moving from the many-channel server host to a genuine dual-channel consumer desktop costs FreeToken only 4% of its decode rate — but costs llama.cpp 20% of its rate, because llama.cpp’s CPU-resident experts are starved by the weaker host memory subsystem in a way FreeToken’s bandwidth-adaptive split isn’t. This is direct evidence that the $q^\star$ policy is doing exactly what it claims: automatically absorbing a weaker host-memory subsystem rather than being blindsided by it.

At the frontier-scale tier, FreeToken serves GLM-5.2 on the single RTX PRO 6000 at 14.9 tok/s against llama.cpp’s 7.3 tok/s (2.0×), with bit-identical expert weights and comparable mean TTFT (7.5s vs. 7.8s). KTransformers, notably, has no servable path at all for this model on this hardware: its methods require 753GB–1.5TB of host-resident experts against only 512GiB of available host memory on the test box, and its CPU kernels don’t support GLM-5.2’s NVFP4 layout to begin with — a genuine capability gap, not just a speed gap.

6. Related work, briefly

The paper positions itself carefully against three adjacent lines of work. Expert offloading and caching (EdgeMoE, Mixtral-offloading, MoE-Infinity, ProMoE, ExpertFlow, FineMoE) all converge on the same host-pool-plus-GPU-cache architecture FreeToken adopts, but differ only in prediction quality — every miss in these systems is still ultimately served by a PCIe transfer, so decode latency stays bounded by the link regardless of how accurate the predictor becomes, while host compute sits idle the whole time. A complementary line (HOBBIT, SiDA, SMoE, Pre-gated MoE) instead reduces transfer volume by relaxing fidelity — lower-precision replicas, skipped low-scoring experts, or a restructured/fine-tuned router — trading some accuracy for bandwidth. Hybrid CPU-GPU execution work (FlexGen, PowerInfer, Fiddler, KTransformers, HybriMoE) enlists the CPU as an actual compute resource rather than a passive weight store, but the division of work between CPU and GPU is typically either fixed at startup or recomputed by host-side heuristics whose per-step scheduling and synchronization cost can’t be captured into a CUDA Graph — which is precisely the gap Section 4’s implementation work closes. Hierarchical memory management systems (SGLang HiCache, WiSP, eLLM, FluxMoE) tier KV cache or expert pages across GPU/host/remote storage but move only passive bytes: when something is absent, it can only be fetched, never computed elsewhere. FreeToken’s stated contribution is combining both levers — an elastic full-expert cache with unified prefill/decode residency, plus the option to compute a missing expert in place rather than only fetching it — coordinated by a single measured-bandwidth model that’s cheap enough to live inside a captured CUDA Graph.

7. Limitations

The paper is reasonably direct about several scope boundaries worth restating explicitly:

• Quality is a non-issue by construction, but that’s also a limitation of ambition. Because FreeToken changes only how the (unmodified) model is executed and scheduled — never approximating, quantizing beyond what the checkpoint already specifies, or altering the routing — there’s no quality-vs-speed trade-off to evaluate, and the paper correspondingly reports zero accuracy/perplexity numbers. This is a genuine strength for a systems paper, but it also means FreeToken cannot, by design, capture the additional speedups available to methods that are willing to trade a small amount of fidelity (the SiDA/SMoE/HOBBIT family in Section 6) — a ceiling the paper doesn’t quantify against.
• The bandwidth measurements are static per-deployment, not continuously re-profiled. $B_P$ and $B_H$ are profiled once on the deployed machine and used to derive $q^\star$; the paper doesn’t discuss what happens if a concurrent application (a game launching mid-session, a browser doing a large download) transiently steals PCIe or DRAM bandwidth away from the serving engine’s actual measured values, which would make the “optimal” $q^\star$ policy compute a stale split for the duration of that contention.
• CPU capability variance is only lightly explored. All test systems have reasonably capable modern CPUs (Core i9, Ryzen 9, or server-class Xeons capped to consumer thread counts); the paper doesn’t test a genuinely weak or older CPU where $B_H$ might be so low that CPU execution is rarely worth doing at all, which would be an interesting edge case for the $q^\star$ formula’s robustness.
• The frontier-scale (753B) demonstration is a single data point. GLM-5.2 on the RTX PRO 6000 is evaluated on one workload (the math agent, per Figure 6’s caption) rather than across all four agentic workloads used for the main RTX 5090 breakdown, so its generalization across workload types is less thoroughly established than the main results.
• No multi-GPU or multi-machine story. Everything in the paper is single-GPU, single-machine. How the bandwidth-adaptive execution model would need to change for a setup with, say, two consumer GPUs sharing one host (a genuinely common “gaming PC with two cards” configuration) is not discussed.

8. Critical analysis

(a) Weaknesses and flaws specific to this paper. First, the $q^\star$ derivation (Section 3.2 / Eq. 3) is a clean closed-form result, but it rests on an implicit assumption that the CPU execution branch and the PCIe transfer branch draw on host-memory bandwidth additively and losslessly — i.e., that $B_H$ (measured presumably with the CPU branch running alone) and $B_P$ (measured presumably with the transfer branch running alone) simply subtract to give the residual $B_R$ available when both run concurrently. In practice, concurrent DRAM access from two independent consumers (a DMA engine and CPU cores) can suffer additional contention effects (page/bank conflicts, NUMA-locality mismatches, memory-controller queuing) that make the true achievable concurrent bandwidth lower than the naive linear-subtraction model predicts. The paper’s strong empirical results suggest this approximation error is small enough not to matter in practice on the tested hardware, but the paper doesn’t isolate or quantify this specific approximation gap directly — a controlled measurement of “predicted $q^\star$-optimal latency” versus “measured latency” at a few different $q$ values would have made the formula’s practical tightness more convincing rather than merely inferred from end-to-end throughput numbers.

Second, the semantic-aware state-checkpoint mechanism (Section 3.1) is motivated with three specific agent harnesses (OpenClaw, OpenCode, SWE-agent) whose context-editing behavior conveniently aligns with special-token boundaries — but the paper doesn’t test what happens with a harness that edits context without respecting these boundaries (e.g., a naive sliding-window truncation that cuts mid-block rather than at a block edge). Given that the entire benefit of semantic anchoring collapses to “no checkpoint survives, fall back to full re-prefill” in that case, it would have strengthened the paper to characterize how gracefully it degrades, rather than only demonstrating the favorable case.

(b) Limitations the authors understate or omit. The paper reports “FreeToken supports more than 20 MoE models” in the introduction and abstract, but the detailed evaluation covers only three model families (DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) — the other 17+ supported models receive no throughput, TTFT, or miss-rate numbers anywhere in the paper. This is a common and generally understandable gap in systems papers (exhaustively benchmarking 20+ models is expensive), but the “20+ models supported” framing in the abstract creates an impression of breadth that the actual evaluation doesn’t substantiate; a reader should treat the quantitative claims as validated for the three tested model families specifically, not for the full supported set. Similarly, the paper doesn’t discuss engine memory overhead or the exact VRAM cost of the CUDA-graph-captured buffers themselves (pinned I/O buffers, fixed-shape work buffers, task descriptors) — on the most memory-constrained tested device (the 8GB laptop), this bookkeeping overhead could matter proportionally more than on a 96GB workstation GPU, and it isn’t quantified anywhere.

(c) Concrete, specific improvement suggestions. (1) Add a direct micro-benchmark isolating the residual-bandwidth model’s prediction accuracy — measure actual concurrent PCIe-transfer-plus-CPU-execution throughput at several $q$ values around the predicted $q^\star$ and compare against the model’s prediction, to directly validate (or bound the error of) the “bandwidths subtract linearly” assumption underlying Eq. 1–3. (2) Report throughput/TTFT/miss-rate numbers — even abbreviated ones — for a representative sample of the other 17+ “supported” models mentioned in the abstract, so the breadth claim and the depth of evaluation are more proportionate. (3) Test the semantic-anchor checkpoint mechanism against at least one agent harness whose context-editing pattern does not align with special-token boundaries, to characterize the graceful-degradation behavior (does it silently fall back to full re-prefill, or does something worse happen?) rather than leaving that failure mode entirely unaddressed.

9. Reproducibility notes

The paper specifies its hardware configurations precisely (six named GPU/CPU/DRAM combinations with measured PCIe and host bandwidths in Table 1), the exact model checkpoints used (including quantization formats — MXFP4 for DeepSeek-V4-Flash, NVFP4 for the laptop’s Qwen3.6 build and for GLM-5.2), and the four agentic workloads with enough detail (harness name, turn count, approximate token counts) to plausibly reconstruct similar evaluation traces. The authors release the system’s code at https://github.com/FlashML-org/FreeToken and a downloadable build at flashml.ai, which meaningfully raises the reproducibility bar relative to a paper describing a closed or internal-only system — a reader can, in principle, verify the core throughput claims on their own hardware rather than taking the reported numbers on faith. The core algorithmic contribution (the $q^\star$ formula, Eq. 1–4 in this review) is fully specified and requires only two empirically measurable numbers ($B_P$, $B_H$) to reproduce on new hardware, which is a genuinely low bar for independent verification compared to methods requiring learned predictors or extensive workload-specific tuning.

10. Conclusion

FreeToken’s contribution is best understood as taking a genuinely edge-specific systems problem — heterogeneous, non-dedicated, wildly variable consumer hardware serving a model whose expert pool categorically exceeds available VRAM — and refusing to solve it with a single fixed policy. Instead, it derives, from two numbers measured on the machine actually in front of it, a closed-form answer to “how should this specific miss be served right now” that adapts automatically across an 8GB laptop and a 96GB workstation GPU alike. The full-layer double-buffered prefill pipeline, the semantic-anchor checkpointing that lets agentic context edits avoid wholesale re-prefill, and the elastic runtime-resizable cache are each individually sensible engineering choices, but the paper’s most quotable systems insight is the residual-bandwidth argument itself: once you notice that PCIe transfer and CPU execution compete for the same underlying host-memory bandwidth, the “how much to offload to which path” question stops being a tuning knob and becomes a two-line formula. The honest scope limits — three model families evaluated in depth despite 20+ claimed support, no direct isolation of the bandwidth-additivity assumption’s error, and no discussion of what happens when an agent harness edits context in ways that don’t respect semantic boundaries — keep this from being a universal answer to edge MoE serving, but they don’t undercut the core result: a personal machine, treated as one elastic platform rather than a small GPU with some extra RAM bolted on, really can serve models it has no business serving.