Pie: Who Should Control the LLM Generation Loop?

Review date: October 8, 2026
Author: Zhongzhu Zhou
Paper reviewed: Pie: A Programmable Serving System for Emerging LLM Applications
Paper authors: In Gim, Zhiyao Ma, Seung-seob Lee, Lin Zhong
arXiv and version: 2510.24051v1, October 28, 2025

1. The useful question is who owns the next step

I read Pie as an argument about the boundary between an application and its serving engine. A conventional completion endpoint asks an application to provide a prompt and a few generation options. The engine then decides how to batch the request, manage its cache, and advance decoding. This works well when the application genuinely wants an uninterrupted sequence of tokens. It becomes awkward when the application wants to stop at a tool call, keep selected history alive, fork a search branch, or change the sampling policy after observing an intermediate result.

Pie moves those decisions into a user program called an inferlet. The program runs near the serving engine and calls primitives for embedding, forward computation, sampling information, memory management, and I/O. A shared control layer still batches work across applications. The idea is therefore more precise than running an agent on the server: the application receives control over the sequence of model operations and the lifetime of model state.

Figure 1 (paper Fig. 2): inferlets issue API calls while a shared control layer batches them. Cropped from Gim et al.; the diagram distinguishes application policy from shared execution.

Three examples make the value concrete. A reasoning search knows which branches are dead and can release their cache. A tool-using agent knows that its history will be needed when an HTTP request returns. A constrained decoder knows which token choices are legal at the current grammar state. The same engine may host all three without adopting one global policy that fits none of them particularly well.

This flexibility has a measurable price. Moving sampling and embedding decisions out of a fused generation loop can interrupt pipelining. Application-managed cache can also reserve memory while a program is waiting on slow external services. The important question is whether application knowledge saves more time or memory than the new control boundary costs.

The paper reports small standard-completion overheads and improvements on several structured and agentic workloads. I will separate those results from three kinds of inference in this review: algebra derived from the architecture, illustrative numerical examples, and proposed experiments that would strengthen the evidence. None of the examples below is a new measurement of Pie.

The reading is pinned to the 16-page arXiv v1 PDF, including its references. Later project releases can change API details and capabilities; they do not retroactively change what this evaluation established. The official project repository is a resource pointer. The technical analysis here uses the paper’s methods, figures, tables, and limitations.

2. Prerequisites: cached state, positions, and resource lifetime

2.1 What a KV cache actually preserves

For an autoregressive Transformer, a new query attends to keys and values created for earlier positions. A KV cache stores those projected states so that every decode step need not recompute the entire prefix. It is a cache of model computation, not merely a copy of the text. Its validity depends on the model, token sequence, position assignment, attention semantics, and any adapter that affects those states.

For a dense cache with $L$ layers, $H_{\mathrm{kv}}$ KV heads per layer, head dimension $d$, $T$ cached tokens, and $b$ bytes per scalar, the storage accounting is:

$$M_{\mathrm{KV}}=2L H_{\mathrm{kv}}dTb. \tag{1}$$

The factor two counts keys and values. Each layer contributes a head-by-dimension array for every token. Multiplying by the scalar size converts element counts to bytes. This is an explanatory model, not a memory formula claimed by Pie, and it excludes metadata, padding, allocator overhead, and temporary buffers.

For illustration, $L=32$, $H_{\mathrm{kv}}=8$, $d=128$, and BF16 storage ($b=2$) give 131,072 bytes per token, or 128 KiB. An 8,192-token history occupies 1 GiB under these assumptions. Keeping ten such independent histories during tool calls can therefore consume about 10 GiB even while none of them is asking the GPU to compute.

Paged allocation replaces one contiguous sequence buffer with independently managed pages. If each page holds $P$ tokens, the number allocated for a sequence is $\lceil T/P\rceil$. The unused token slots in the last page are:

$$W(T,P)=P\left\lceil T/P\right\rceil-T, \qquad 0\le W(T,P)<P. \tag{2}$$

Paging reduces the need to reserve a large maximum-length buffer, but does not make retained tokens free. Pie uses the paged abstraction and describes KvPage sizes of 8–32 tokens. Some paper examples simplify this to one token per page; that simplification must not be read as the physical configuration.

2.2 Prefill, decode, and the cost of forgetting

Prefill computes states for an input sequence, often with substantial parallelism. Decode advances the sequence using its existing cache. If an agent discards the cache when it calls a tool, its next completion request may replay a growing conversation. Automatic prefix caching can recover some of that work, but only if the required states remain resident and match the new request.

Consider $R$ rounds, an initial prompt of $P_0$ tokens, and $u$ additional tokens introduced before each later round. A simple token-accounting example gives:

$$C_{\mathrm{replay}} =\sum_{r=0}^{R-1}(P_0+ru) =RP_0+\frac{uR(R-1)}{2}, \qquad C_{\mathrm{retain}}=P_0+u(R-1). \tag{3}$$

The second expression assumes all previous state remains valid and resident. It counts newly prefilling text, not autoregressive generation work. The difference grows with rounds because replay repeatedly processes old input. Real latency is not proportional to these counts in general: batching, prompt length, kernel efficiency, and cache hits all intervene.

Figure 2 (original analytical illustration): cumulative prefill-token accounting from Eq. (3), with a 1,000-token initial prompt and 200 newly introduced tokens per round. This is not a measured speedup or a workload taken from the paper.

The architectural opportunity is to preserve exactly the state the application expects to reuse. The corresponding danger is to keep everything forever. A programmable cache interface makes both policies possible.

3. The programming contract: resources, queues, and futures

Pie separates application, control, and inference layers. Inferlets execute in isolated WebAssembly instances. The control layer resolves resource requests, routes non-GPU operations, schedules GPU work, and delivers completion events. The inference layer holds physical resources and executes batched handlers. An inferlet controls policy through handles; it does not receive arbitrary addresses into GPU memory.

The paper exposes two central resource types. Embed holds token embeddings or intermediate output representations. KvPage holds a page of attention state. Allocation returns an opaque handle within the inferlet’s virtual resource address space. Import and export operations allow deliberate sharing across programs, with the control layer maintaining the mapping to physical memory.

This separation is a substantive design choice. Giving every application unrestricted physical pointers would make composition fast to express but difficult to isolate or relocate. Returning a virtual handle permits the engine to retain authority over allocation and access. It does not by itself specify a complete authorization scheme for sharing, nor does it guarantee fairness under memory pressure.

A command queue makes ordering explicit. GPU-bound operations reference a queue; commands within it have ordered dependencies, while distinct queues can expose independent work. Asynchronous calls return futures where a result must eventually reach the inferlet. A single-threaded event-driven program can therefore have multiple operations in flight without being a multithreaded program.

A useful mental model is to distinguish three events: enqueue, execute, and observe. Enqueuing a forward call does not mean the output is ready. Execution may happen after batching with other applications. Observation requires waiting for the relevant completion. Resource cleanup must respect this ordering too; otherwise an application would attempt to recycle storage still used by pending work.

The API groups operations into traits such as Allocate, Forward, InputText, InputImage, Tokenize, and OutputText. Traits can require other traits. An inferlet can discover model capabilities before deciding which path to take. The paper describes 42 APIs, including 18 for model execution and resource management. That count describes the paper’s interface, not a promise about every later release.

The forward interface is particularly important. It receives input embeddings, existing KV pages, requested output locations, and optionally an attention mask. Sequence positions are explicit. This allows a program to change which stored tokens a future query can attend to. It does not mean that deleting one old token recomputes all later keys and values that were originally influenced by it.

For example, suppose token C was computed while attending to A and B. Removing B from the visible cache for a future token D prevents D from attending directly to B. C’s stored state may still encode information from B. Thus cache masking and full counterfactual recomputation are different operations. This distinction matters when interpreting attention experiments or proposing application-specific eviction.

Pie deliberately leaves kernel fusion, quantization, and tensor-parallel execution behind the inference-layer boundary. Its claim is that generation policy can become programmable without making each application implement GPU kernels. The complementary cost is that a new model capability may require a new handler or trait before an inferlet can use it. A programmable interface is only as expressive as the information and actions exposed through that interface.

4. Reconstructing a generation loop without hiding its dependencies

Algorithm 1. Autoregressive generation with explicit state

The following numbered pseudocode reconstructs the paper’s programming model. Names are explanatory rather than a copy-paste API recipe. The cache is provisioned here for simplicity; incremental allocation is a separate policy choice.

01  Check that the model supports text input and output.
02  Create queue q; tokenize the prompt into token IDs.
03  Allocate input embeddings, one output slot, and KV pages.
04  Enqueue embedding of prompt IDs at their true positions.
05  Enqueue prompt forward pass, saving KV and last output.
06  Repeat until the output limit is reached:
07      Await the next-token distribution from the output slot.
08      Apply the selected sampling rule; obtain token x.
09      Emit x; stop if x satisfies the stopping rule.
10      Embed x at its next sequence position.
11      Enqueue forward using valid history and x's embedding.
12      Save x's KV state and the next output representation.
13  Await outstanding uses of the resources.
14  Release embeddings and pages that are no longer shared.

The sequence explains why a token generator is not simply a loop over a matrix multiplication. Input positions and attention state have to remain aligned. Sampling must observe the correct output representation. A sampled token must be embedded before its state can be appended. The application needs a stopping rule, and resource lifetimes must cover pending operations.

The distinction between predicted and processed tokens is easy to miss. The model predicts the next token from the current last hidden state. Only after that token is fed through the model does its own KV entry exist. A program that increments its cache length immediately after sampling, before the forward pass completes, would describe state it has not yet produced.

The paper’s support library offers a higher-level Context abstraction that hides much of this routine management. This is essential for usability. Most applications should express a branch, a retained conversation, or a constrained output policy rather than manually track every embedding buffer. The low-level interface is valuable as a foundation for those libraries and for unusual policies.

There are three possible levels of control. A high-level completion endpoint minimizes application responsibility. A Context library exposes structured operations while retaining common invariants. Direct inferlet API calls maximize flexibility but transfer more correctness obligations to the developer. None of these levels is universally best; the appropriate level depends on whether a workload’s unusual structure actually creates an optimization opportunity.

A simple correctness test at the conceptual level is to compare a monolithic prompt forward with a split prefix-forward plus continuation-forward. They should represent the same causal computation when model parameters, positions, masks, and numerical conventions agree. This is an explanation of the contract, not a report of a test performed here. Deviations such as windowed masking intentionally change that contract and require a separate quality argument.

Error handling also belongs in the application contract. If a tool fails, the inferlet may retain state and retry, fall back to another tool, or finish with an error. If the inferlet is terminated for resource pressure, the application may lose its in-memory history. A production design needs to define which of these failures are recoverable and which external effects may already have occurred.

5. Batching API calls while preserving meaning

A programmable engine cannot give every inferlet exclusive GPU execution. It must combine work from applications with different control flows. Pie uses horizontal batching across queues and vertical batching of compatible consecutive operations within a queue. The second term should not be confused with pipelining across Transformer layers.

Figure 3 (paper Fig. 4): commands of the same type can share a dispatch across queues, or within a queue when their dependencies can be respected. Cropped from Gim et al.

Batch eligibility is a semantic property before it is a performance choice. As a conservative illustration, let $R(a)$ and $W(a)$ be the resource sets read and written by operation $a$. Two operations are safely independent if:

$$W(a)\cap\bigl(R(b)\cup W(b)\bigr)=\varnothing, \qquad W(b)\cap\bigl(R(a)\cup W(a)\bigr)=\varnothing. \tag{4}$$

These conditions exclude write-write and read-write hazards. They are sufficient for independent parallel execution, but not necessary for every legal fused execution. A specialized handler can preserve an ordered dependency internally. The paper, for example, notes that a prefix forward and its continuation can be combined by vertical batching. One should not equate “batched” with “unordered.”

Commands of different operation types cannot automatically share a handler. Embedding and forward computation require different execution paths. Pie forms eligible batches by API type, orders commands by queue priority within a batch, and truncates at the backend’s maximum supported batch size. Between eligible types, it chooses the batch containing the oldest pending call.

Algorithm 2. Work-conserving API dispatch

This pseudocode summarizes the scheduling policy described in Sections 5.2 and 6.1 of the paper. Compatibility and dependency checks are abstract predicates, not a claim that the paper specifies every scheduling corner case.

01  On an incoming API call, identify its queue and dependencies.
02  Handle control-only calls through the control layer.
03  Enqueue GPU-bound calls; keep their ordering information.
04  If the GPU is busy, keep collecting incoming calls.
05  On a GPU-idle notification:
06      Find eligible calls for each handler type.
07      Combine compatible calls across command queues.
08      Include eligible vertical groups with preserved semantics.
09      Order each candidate batch by queue priority.
10      Truncate candidates to the backend batch limit.
11      Dispatch the candidate with the oldest pending call.
12  Deliver completion events and make dependents eligible.

The dispatch trigger is intentionally simple. While the GPU is busy, arrivals accumulate. When it becomes idle, the controller immediately forms the next batch. This is work-conserving because the GPU is not deliberately held idle waiting for a larger batch when executable work exists.

Why not always dispatch immediately? Small dispatches sacrifice amortization and parallelism. Why not wait for exactly $K$ calls? At low arrival rates, a queue may wait a long time to reach $K$. A timer limits waiting but chooses one time scale for workloads whose service times vary. Pie uses execution itself as an accumulation interval.

Work conservation does not establish optimal latency or fairness. A large forward batch can delay a short embedding request. Priority ordering can favor one queue repeatedly. Oldest-call selection across operation types does not necessarily imply per-user fairness. These are policy boundaries that a throughput measurement alone cannot resolve.

6. Tool execution becomes part of the same stateful program

Pie is particularly compelling when inference alternates with external work. In a remote orchestration design, the client receives a tool-call request, invokes the tool, updates the transcript, and submits another completion. In Pie, the inferlet can invoke network operations while keeping its model state. The external service still takes time; what changes is the placement of orchestration and the lifetime of context.

Figure 4 (paper Fig. 5): the two deployments place tool-handling logic on different sides of the client-server boundary. Cropped from Gim et al.; moving the handler does not eliminate the tool’s own latency.

A first-order latency decomposition helps identify when this matters. Let $G$ be shared model work, $I$ unavoidable external-service time, $m$ eliminated orchestration round trips of latency $r$, $C$ avoidable context-recomputation time, $N$ generated tokens, and $\delta$ the additional per-token control cost. Let $L$ be launch overhead and $S$ additional scheduling or bookkeeping cost. Then an explanatory comparison is:

$$\begin{aligned} T_{\mathrm{remote}}&=G+I+mr+C,\\ T_{\mathrm{inferlet}}&=G+I+N\delta+L+S,\\ T_{\mathrm{remote}}-T_{\mathrm{inferlet}} &=mr+C-N\delta-L-S. \end{aligned} \tag{5}$$

This isolates the break-even condition: retained state and avoided round trips must outweigh added control cost. It assumes the shared terms are comparable; real batching can change $G$, and overlapping tool execution can change the critical contribution of $I$.

For a deliberately simple example, eight avoided round trips at 20 ms save 160 ms. An added 1.5 ms over 200 generated tokens costs 300 ms before launch overhead. Round-trip removal alone would lose in that example. If avoiding repeated prefill saves another second, the result reverses. These invented inputs illustrate a boundary; they are not the paper’s benchmark configuration.

Algorithm 3. A stateful tool-using inferlet

The application-specific optimizations in Section 7.2 can be organized as follows. Early execution is conditional on application semantics; it is not valid for arbitrary tool calls.

01  Load the task and the allowed tool descriptions.
02  Import any valid, authorized reusable prefix state.
03  Generate incrementally while retaining the conversation KV.
04  Detect a complete, validated tool name and argument set.
05  If the call is independent and safe to launch early:
06      Start it asynchronously and retain its completion future.
07  Otherwise wait until the generation dependency is resolved.
08  Await results only where later computation needs them.
09  Add required observations and continue generation.
10  If a tool description is provably no longer needed:
11      Apply the chosen context-retention policy.
12  Release task-private resources; export only reusable state.

The paper considers a workflow where some tool descriptions recur frequently, most calls are fire-and-forget, and most tools are used only once. Those properties motivate three optimizations: cache frequent descriptions, overlap calls with generation, and mask no-longer-needed specifications.

The word “provably” in the pseudocode is my caution about semantics. A tool may be called twice after a failure. Generated arguments may remain incomplete until a closing delimiter. A fire-and-forget operation can still have a consequential external effect. Useful application knowledge must be expressed as an explicit contract; otherwise a faster workflow may simply be doing a different job.

This section also explains the distinction between local control and global resource authority. The inferlet can decide that a history is useful. The platform must decide whether preserving that history is affordable while other tenants wait. Pie provides a place to express the first decision; its current contention policy only partially addresses the second.

7. Sampling access has an information budget

Returning a vocabulary-wide probability distribution to an inferlet can be expensive. The paper describes truncating the returned distribution to a configurable top-$K$, with a default $K=256$. This is a sensible data-movement optimization, but it creates a semantic question: what information does the application need to make its next decision?

Let $p(v)$ be the full next-token distribution and $S$ the returned top-$K$ set. Its retained mass is $Z=\sum_{v\in S}p(v)$, and its omitted mass is $\epsilon=1-Z$. If a client simply normalizes the returned probabilities, it samples from:

$$q(v)= \begin{cases} p(v)/Z,&v\in S,\\ 0,&v\notin S. \end{cases} \tag{6}$$

Assuming $Z>0$, the total-variation distance from the original distribution is:

$$\begin{aligned} \operatorname{TV}(p,q) &=\frac12\left( \sum_{v\in S}\left|p(v)-p(v)/Z\right| +\sum_{v\notin S}p(v)\right)\\ &=\frac12\left((1/Z-1)Z+\epsilon\right) =\epsilon. \end{aligned} \tag{7}$$

This derivation is mine, not a theorem in the paper. It does not assert that Pie always samples from $q$ or that every user wants to preserve $p$. Top-$K$ sampling may itself be the desired policy. The point is that an API cannot generally discard probability mass and still promise every possible full-distribution operation without a fallback.

Greedy selection is straightforward if the true maximum is retained. Grammar constraints can be harder. Suppose every returned token is illegal in the current grammar state, while a legal token lies outside $S$. A mask applied only to the returned set cannot recover that token. Watermarking and distribution-level checks may also require more than a small candidate list.

Several alternatives follow. The application can request a larger $K$. The system can expose a filtered-distribution handler that applies constraints before truncation. A fused sampling trait can run custom logic near the logits while returning only the chosen token. Each alternative trades generality, data movement, and trust in a different way.

The paper’s design supports extensible traits, which gives these alternatives a natural place to live. However, proposing such a trait is different from demonstrating that it preserves the performance and isolation properties of the existing system. An interface extension can introduce a new bottleneck or expose information that was previously hidden.

Attention introspection has the same structure. An inferlet cannot implement an attention-score-dependent eviction policy if the forward handler never returns those scores. The paper explicitly discusses extending Forward for this purpose. Programmability is therefore not unrestricted access to the model; it is the ability to compose a defined set of capabilities.

I find this boundary useful because it prevents an overly broad reading of the paper. Pie can make many policies user-programmable, but adding every imaginable policy may require further system support. The question for each proposed optimization is which intermediate values it needs, where those values live, and whether moving them across the interface costs more than the optimization saves.

8. Reading the application results with their conditions attached

The evaluation uses a GCP g2-standard-32 instance with an NVIDIA L4 and 24 GB of GPU memory. The paper describes BF16 weights, activations, and KV state for Llama-family models of 1B, 3B, and 8B parameters. The comparison includes vLLM 0.6.0, SGLang 0.4.4, and relevant specialized systems. Pie, vLLM, and SGLang use FlashInfer to reduce differences caused solely by GPU kernels.

End-to-end latency is measured from a remote Python client on a campus network. This matters because eliminating client-server orchestration round trips is one of the mechanisms under examination. A deployment with orchestration already colocated beside the engine may realize a smaller benefit. Conversely, a longer network path can amplify that component without improving GPU efficiency.

Figure 5 (paper Fig. 6): latency and throughput are normalized separately within each workload. Cropped from Gim et al.; normalized bar heights should not be compared as absolute seconds across different agents.

For a 1B model, the authors evaluate ReACT, CodeACT, and Swarm with 8, 8, and 32 external I/O operations per agent respectively. The reported Pie latency and throughput are:

Workload	Latency	Throughput	External I/Os
ReACT	4.27 s	29.94 agents/s	8
CodeACT	3.18 s	40.18 agents/s	8
Swarm	6.14 s	5.21 agents/s	32

The paper attributes gains on smaller models largely to avoiding orchestration round trips, and gains on larger models increasingly to retaining KV state instead of repeating prefill. It also reports little performance difference with fewer than two external interactions in the studied setting. That is a useful boundary: not every agent application necessarily benefits.

The latency and throughput values must not be treated as reciprocals. Throughput aggregates concurrent work; latency follows a single workflow. A system completing 30 agents per second can still take several seconds per agent when many agents overlap. The missing configuration details for a particular result should not be filled in by assuming one sequential client.

8.1 The larger custom-workflow gain is a composition result

Figure 6 (paper Fig. 7): caching frequent specifications, launching suitable calls early, and masking single-use specifications are added successively. Cropped from Gim et al.; the paper reports a 3.5× throughput gain for this tailored workflow.

The 3.5× result is especially interesting because it is a gain from application semantics, not a claim that every forward pass becomes faster. It should also be read narrowly. The three favorable properties of the workflow were selected explicitly. A service with rarely reused descriptions, dependent tools, or repeated recovery calls may not permit the same transformations.

The stacked experiment is not a full factorial ablation. If optimization C is measured only after A and B, the marginal effect of C is conditional on those earlier choices. One cannot add the bar increments as though each were an independent benefit that transfers unchanged to another workload.

A stronger follow-up would evaluate all combinations and report output-quality equivalence, failed tool calls, memory occupancy, and tail latency. That would distinguish improved orchestration from changed computation or reduced attention context. It would also show whether a policy that helps one inferlet harms other inferlets by retaining scarce memory.

9. Expressiveness evidence is different from a universal speed ranking

The paper demonstrates attention variants, structured decoding, prompt-lookup speculative decoding, beam search, prefix trees, and several reasoning workflows. This breadth supports the central architectural claim: many generation strategies can be expressed above a common set of resource and execution primitives.

Figure 7 (paper Fig. 8): comparisons across supported workloads. Cropped from Gim et al.; crosses indicate unsupported techniques or unavailable comparisons, rather than a measured zero throughput.

The use of common kernels in the main baselines is a meaningful control, but it does not eliminate all differences. Systems can vary in orchestration placement, attention semantics, batching policies, cache retention, or the exact version of an algorithm. Each comparison needs to identify which of these differences is intended to be the treatment.

The paper’s speculative-decoding experiment uses n-gram prompt lookup. That result should not be generalized to every learned drafter or every tree-verification scheme. Likewise, the absence of a matching method in a particular SGLang version does not establish that the current project cannot implement speculative decoding. These are time- and method-specific support comparisons.

For attention sinks, the paper reports about 1.5× lower latency and more than 30× higher throughput relative to the specialized StreamingLLM implementation. It explicitly notes differences in the underlying GPU kernel libraries. I would use this as evidence that Pie can express the technique with an efficient backend, not as an isolated measurement of the benefit of programmability.

The deliberate prompting workloads are simplified versions of the original applications. They include arithmetic-style reasoning and document summarization, with recursive branching up to depth five for one configuration. The reported improvements, up to 28% lower latency and 34% higher throughput in this group, are useful evidence about serving those workloads. They are not demonstrations of superior reasoning accuracy.

A practical way to read the figure is in three passes. First, ask whether a method is supported at all. Second, compare performance only among implementations that perform equivalent work. Third, ask why the difference appears: cache reuse, fewer remote round trips, improved kernels, or altered context. This avoids collapsing expressiveness, application optimization, and raw backend speed into one ranking.

Code length is also only one part of expressiveness. Table 2 reports small inferlets for many examples, and a shared support library supplies common machinery. That suggests policies can be packaged economically. It does not establish the time required to debug resource lifetimes, validate new masks, or maintain compatibility as traits evolve. A few lines of high-level code can stand on a substantial stack of invariants.

The most convincing interpretation is that Pie offers an extensible point at which specialized strategies can coexist. The evidence is weaker for the stronger claim that the same interface will always retain comparable performance as models, workloads, and hardware scale. That stronger claim needs broader experiments, especially where GPU execution gets fast enough that control overhead becomes prominent.

10. The overhead is mainly lost overlap, not simply WebAssembly

The microbenchmarks make an important distinction between a runtime boundary and the opportunity cost of an interface. A cheap API call can still force two expensive stages to run sequentially when a monolithic engine would overlap them.

For the 8B case in Table 3, the reported added components include 1.320 ms from losing pipelined sampling, 0.070 ms from losing pipelined embedding, 0.050 ms for batch scheduling, and 0.070 ms for returning the distribution. Boundary crossing and Wasm processing are listed at much smaller scales: 0.006, 0.001, and 0.001 ms.

Adding the displayed entries gives:

$$\Delta_{\mathrm{listed}} =1.320+0.070+0.050+0.070+0.006+0.001+0.001 =1.518\ \mathrm{ms}. \tag{8}$$

The reported end-to-end TPOT difference is $65.59-64.06=1.53$ ms. The 0.012 ms gap between these displayed totals is small; rounded or separately measured components need not sum exactly. The evidence nevertheless strongly locates the main cost in lost pipeline overlap rather than the sandbox itself.

Table 4 makes the model-size effect concrete:

Parameters	vLLM TPOT	Pie TPOT	Added time
1B	16.83 ms	18.75 ms	1.92 ms
3B	30.30 ms	32.01 ms	1.71 ms
8B	64.06 ms	65.59 ms	1.53 ms

The relative penalty follows directly from the denominator:

$$\rho=\frac{T_{\mathrm{Pie}}-T_{\mathrm{base}}}{T_{\mathrm{base}}}, \quad \rho_{\mathrm{1B}}\approx11.41\%, \quad \rho_{\mathrm{8B}}\approx2.39\%. \tag{9}$$

This trend is a statement about the tested configurations. It should not become a universal law that larger models always make control overhead negligible. Faster hardware, different batching, quantization, or a different context length can change the denominator and the amount of overlap available.

Figure 8 (paper Fig. 10): control-layer and inference-layer API latency as concurrency increases. Cropped from Gim et al.; the microbenchmark excludes handler processing and disables batching.

The prose reports control-layer calls below 30 microseconds even with 896 concurrent inferlets, although the highest plotted point in Fig. 10 appears closer to 40 microseconds. I therefore treat the result as a tens-of-microseconds scale observation, not a strict 30-microsecond bound. Inference-layer calls range from 10 to 300 microseconds as concurrency increases, with Python deserialization identified as a major contributor. These numbers cannot simply be multiplied by an arbitrary call count to predict end-to-end latency: asynchronous calls overlap, dependencies differ, and batching was disabled for the measurement.

Startup is another separate quantity. Figure 9 reports roughly 10–50 ms for warm launches and 35–81 ms for cold launches at the studied concurrency levels. An 80 ms start may be small for a long reasoning job and material for a very short interactive response. Describing it as universally negligible would hide that distinction.

If launch costs $L_0$ and token generation costs $t$ per token for an output of length $N$, its share of a simplified runtime is:

$$f_{\mathrm{launch}}=\frac{L_0}{L_0+Nt}. \tag{10}$$

With illustrative values $L_0=40$ ms and $t=20$ ms, this share is 11.1% for 16 tokens and about 0.39% for 512 tokens. Again, these are derived examples, not benchmark results.

Figure 9 (redrawn from paper Tables 4 and 5): measured TPOT values and batching throughput. The numerical values are retained; the bar-plot presentation is new.

The batch-policy table deserves a separate arithmetic check. Its Eager, K-only, T-only, and Adaptive rates are 5.61, 30.09, 78.11, and 84.85 requests/s. These yield:

$$\frac{84.85}{5.61}\approx15.12,\qquad \frac{84.85}{30.09}\approx2.82,\qquad \frac{84.85}{78.11}\approx1.086. \tag{11}$$

Thus the table implies about 15.1× versus Eager, 182% more than K-only, and 8.6% more than T-only. The adjacent prose says up to 17× and 8–40% against the fixed policies. Those claims do not follow directly from the displayed table. I retain the table values and flag the discrepancy rather than invent an unreported configuration to reconcile it.

11. Why these design choices, and when alternatives win

Why colocate control? The application can preserve state and react without repeatedly involving a remote client. The alternative is a client-side framework using richer serving endpoints. That alternative is attractive when most work is external, when the engine already preserves sessions, or when operators do not want user programs near the inference service. Equation (5) identifies the relevant costs instead of presuming colocation always wins.

Why expose pages? The application knows semantic reuse opportunities that a global LRU policy may miss. The alternative is to expose only high-level cache hints, such as retain-until or shared-prefix identifiers. Hints reduce programming burden and preserve more optimization freedom for the engine, but may not express a novel attention pattern. A sensible library can use high-level defaults while leaving lower-level operations available for policies that need them.

Why WebAssembly? The paper prioritizes isolation, portability, and lightweight startup for many inferlets. Native processes, containers, and embedded interpreters offer different tradeoffs. The comparison is not a theorem that Wasm is the safest or fastest option for every deployment. More importantly, runtime memory isolation does not itself provide network authorization, cache-sharing permissions, or a complete defense against resource exhaustion.

Why a layered backend? Separating control from GPU execution makes specialized kernels reusable across many policies. The alternative is a monolithic optimized loop with extension hooks. Hooks can be excellent when the needed policy fits their scope. Pie’s more granular API pays additional control costs in return for allowing combinations that fixed hooks may not anticipate.

Why work conservation? It avoids idle GPUs when eligible work exists and uses execution time to gather new requests. Alternatives include latency-aware batching, workload classification, or deadline-based scheduling. These can improve tail behavior, but require estimates and policies that the simple design avoids. The paper’s experiment supports its adaptive policy on one saturated setting; it does not solve general SLO-aware scheduling.

There is also a useful boundary between policy innovation and new hardware primitives. If an application merely reorders existing operations or retains known cache pages, an inferlet may suffice. If it needs a new attention statistic, a different model architecture, or a new fused operation, the backend may still need to evolve. The trait mechanism organizes that evolution; it does not remove it.

A fused forward-and-sample handler illustrates the tension. It may recover lost overlap for a common decoding case. But if every special policy demands another fused handler, the engine can gradually accumulate the complexity the programmable design sought to avoid. The durable architectural question is how to preserve a small, composable contract while providing efficient fast paths for frequent patterns.

My preference would be to evaluate fast paths against explicit semantic contracts. A fused path is safe when it preserves the resource effects and observable outputs of the granular path for the relevant policy. A transparent fallback should remain available when an application crosses that contract. This is a design suggestion arising from the paper’s tradeoff, not a feature established by its measurements.

12. Limitations: local flexibility meets shared scarcity

The paper explicitly identifies security, centralized control, and resource contention as unfinished areas. These limitations are connected. A tenant can hold memory while waiting for network I/O; that affects scheduler capacity and creates an opportunity for denial of service even if the program is correctly sandboxed.

The current contention policy is coarse: the controller gives earlier inferlets priority and terminates the most recently created inferlets until resources are available. This is simple, but it can waste work and penalize new arrivals. It also makes latency statistics conditional on which jobs survive. A production evaluation should include termination rate, successful-job goodput, and the work spent on jobs that never complete.

Memory pressure depends on residence time as well as allocation size. If class $j$ arrives at rate $\lambda_j$, retains $m_j$ bytes per active request, and has average state lifetime $\tau_j$, a steady-state accounting approximation is:

$$\mathbb{E}[M_{\mathrm{resident}}] \approx\sum_j \lambda_j\,\mathbb{E}[m_j\tau_j]. \tag{12}$$

This uses the intuition of Little’s law applied to retained state; it assumes a stable workload and appropriate averaging. The expectation of the product matters when long requests also use more memory. Replacing it with the product of means can understate pressure when size and lifetime are positively correlated.

The implication is counterintuitive but important: preserving cache can reduce compute while reducing the number of users that fit simultaneously. A policy should compare the expected recomputation saved with the occupancy cost of keeping state. The paper opens this policy space but does not provide a comprehensive resource market or admission-control solution.

Distributed scaling adds another layer. The evaluated controller feeds a single inference backend. Across nodes, a virtual page handle must correspond to state somewhere, and execution placement must account for that location. Migrating state costs bandwidth and may compete with model communication. A global scheduling decision can be locally suboptimal if its chosen worker lacks the necessary history.

Failure recovery is especially subtle with external I/O. If an inferlet performs a tool action and fails before recording completion, retrying may repeat the action. If it instead discards history to recover quickly, the model may behave differently. A robust system needs clearly defined completion, cancellation, and replay semantics. The paper does not establish a solution to these distributed workflow problems.

The evaluation scope is also narrow relative to the proposed abstraction’s breadth. A single L4, models up to 8B, and a set of chosen workflows are enough for a credible architectural proof of concept. They do not demonstrate cluster-scale placement, large-model tensor parallelism, heterogeneous accelerator scheduling, or the tail behavior of a mixed production workload.

Finally, explicit state manipulation can affect model quality. Masking old context or changing the visible token set may improve throughput while changing predictions. Performance and semantic validity therefore need to be evaluated jointly. This is most pressing for application-specific context removal, not for every ordinary cache-preserving operation.

13. Critical analysis: the missing unit is useful completed work

I find Pie’s architectural argument stronger than any single headline speedup. It creates a useful point of separation between application knowledge and shared execution. The remaining question is how to measure the value of that separation when applications compete, fail, and change their own computation.

13.1 Resource ownership needs a pricing signal

Explicit allocation makes state ownership visible, but visibility alone does not cause economical behavior. An inferlet benefits from retaining its own cache while imposing part of the cost on other tenants. Killing the newest program is a blunt response to that externality.

A concrete next step would be a retention interface with a byte budget, expiration time, and estimated reuse value. A policy could prioritize pages according to expected recomputation savings per byte-second. That estimate will be imperfect, so it should be compared with simple baselines such as per-tenant quotas and bounded retention windows. The goal is to reduce wasted compute without converting GPU memory into an unbounded waiting room.

This would also clarify the application-platform contract. The application can state that a page is valuable; the platform can revoke residency under a documented policy. A virtual handle should then have a defined behavior on restoration or failure, rather than leaving developers to infer whether an exported page is a durable object.

13.2 Latency comparisons should remove confounders one at a time

The remote-client setup is appropriate for demonstrating round-trip costs, but it bundles two improvements: moving orchestration and exposing fine-grained state control. I would compare three configurations: remote orchestration with a conventional endpoint, colocated orchestration with the same endpoint, and inferlets with explicit state management. Holding the kernels and application logic constant would reveal each contribution.

For cache retention, vary prefix-cache hit rate and tool latency independently. A baseline that hits its prefix cache almost perfectly is a different comparison from a baseline that recomputes everything. A tool that takes seconds creates a different memory-residence problem from one that takes a few milliseconds. Reporting only aggregate throughput leaves those mechanisms entangled.

13.3 The tailored workflow needs semantic and factorial ablations

The 3.5× example is a useful existence proof, but its assumptions deserve direct stress tests. Increase the probability that a tool is reused. Introduce delayed, failed, or dependent calls. Measure answer quality after masking specifications. Compare each optimization alone and in combination, using identical generated work where the semantics permit it.

The key outcome should be useful completed agents under a latency target, not merely agents/s for one favorable trace. This would expose tradeoffs among speed, context quality, memory pressure, and terminated jobs. It would also prevent an early-call optimization from appearing beneficial merely because it executes actions that later turn out to be unnecessary.

13.4 Some reported numbers need clarification

Two concrete inconsistencies limit precise reuse of the microbenchmark results. First, the ratios computed from Table 5 differ from the adjacent prose, as Eq. (11) shows. Second, Figure 11’s caption describes beam width five, while the explanatory paragraph describes width three when discussing API calls per output token. These do not invalidate the architecture, but they leave the exact interpretation of those measurements uncertain.

I would not silently pick one beam width or reconcile the throughput discrepancy with invented data. A revised table with configuration identifiers, uncertainty, and a consistent definition of “output token” would resolve much of the ambiguity. Beam search is particularly sensitive to the denominator because discarded branches consume work without contributing final-output tokens.

13.5 A compact evaluation matrix would be more informative

A useful extension would vary model size, network round-trip time, context length, external wait time, and concurrency. At each point, report median and tail latency, completed-job goodput, termination rate, GPU busy time, and KV byte-seconds. Add quality checks only where the application changes attention or sampling semantics.

This matrix would test the central hypothesis directly: programmability is valuable when semantic knowledge saves enough work to cover the cost of exposing control. It would also identify the cases where a conventional fused completion loop remains the better choice. That is a more actionable result than treating a programmable architecture as an unconditional replacement for every serving engine.

14. Conclusion

Pie makes the generation process itself programmable. Inferlets can decide how to preserve KV state, interleave tools, branch computation, and interpret outputs, while a shared controller still batches execution. The paper shows that this can preserve competitive standard-task performance and enable useful application-specific optimizations.

The most important cost is not simply running WebAssembly. It is giving up some overlap and transferring responsibility for resource lifetime and semantic correctness to applications. The most important open problem is how local control composes under global scarcity: memory, scheduling fairness, failure recovery, and distributed placement.

My takeaway is a conditional one. When an application knows reuse and dependency structure that the serving engine cannot infer, exposing a programmable contract can turn that knowledge into efficiency. The contract must still make its costs and boundaries visible. Pie supplies a persuasive starting point, while its own limitations and numerical ambiguities make clear which claims require further evidence.

References and figure provenance

1. In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A Programmable Serving System for Emerging LLM Applications, 2025. Primary source for the reviewed architecture and reported measurements; Sections 4–8, Tables 1–5, and Figures 2, 4–11 are the main anchors.
2. Pie project. Official resource entry linked by the paper.
3. The original paper is distributed under CC BY 4.0. Figures 1 and 3–8 in this review are attributed crops of its figures. Figure 2 is an original analytical illustration; Figure 9 redraws published table values. Cropping removes surrounding page text, while the plotted content is unchanged.
4. Equations (1)–(12), numbered algorithms, worked examples, and the critical evaluation proposals are explanatory reconstructions or analysis by this review’s author. They are not additional experiments or formal claims attributed to the paper.