Tag

#LLM Serving

51 posts tagged with this label. Back to all tags or the main feed.

2026

09-05 EN

TreeWY: Removing the Memory Wall in Speculative Decoding for Gated DeltaNet Hybrids
09-05 中

TreeWY 阅读笔记：用树结构 WY 变换拆掉 Gated DeltaNet 混合模型投机解码里的内存墙
09-02 EN

Verification-Aware Training: Teaching Speculative Decoding Drafters to Care About the Rejection Point
09-02 中

验证感知训练（VAT）：让投机解码的草稿模型学会关心「拒绝点」
08-23 EN

FreeToken: Turning Your Gaming PC Into a Frontier-Model Serving Box
08-23 中

FreeToken 阅读笔记：把家用主机变成能跑千亿参数模型的推理平台
08-22 EN

CacheRoute: Why Your LLM Load Balancer Is Fighting Your KV Cache
08-22 中

CacheRoute 阅读笔记：为什么你的负载均衡器一直在跟 KV Cache 打架
08-16 EN

vToken: Turning Token-Level KV Eviction into Actually Reusable GPU Memory
08-16 中

vToken 阅读笔记：如何把 Token 级 KV 淘汰真正变成可用显存
08-15 EN

SwiftQK: When Normalization Becomes the Bottleneck of Tensor Parallelism
08-15 中

SwiftQK：当归一化层成为张量并行的瓶颈
08-12 EN

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
08-12 中

OasisKV 阅读笔记：把投机采样的草稿 token 变成免费的 KV Cache 预取信号
08-07 EN

SALT: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
08-07 中

SALT 阅读笔记：子空间对齐质心-残差训练如何让超低秩 LoRA 服务成为可能
08-06 EN

Fine-Grained Compute-Communication Overlap for MoE: Hiding All-to-All Behind GEMM with Tile-Level Signaling
08-06 中

MoE 计算-通信细粒度重叠：用 Tile 级信号把 All-to-All 藏在 GEMM 背后
07-22 EN

AdaFlash: Taming Variance in Diffusion Speculative Decoding
07-22 中

AdaFlash 阅读笔记：如何驯服扩散式投机解码中的方差问题
07-08 EN

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
07-08 中

DSpark:半自回归生成与置信度调度的投机解码
06-28 EN

Moebius: Seamless Runtime Parallelism Switching for MoE LLM Serving
06-28 中

Moebius：为 MoE 大模型推理服务实现无缝运行时并行策略切换
06-25 EN

ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
06-25 中

ReMP：LLM 推理服务中的低停机运行时并行拓扑重配置
06-21 EN

Tutti: GPU-Centric SSD-Backed KV Cache That Finally Makes SSDs Practical for Long-Context LLM Serving
06-21 中

Tutti 阅读笔记：GPU 原生 SSD KV 缓存，让 NVMe 固态硬盘真正可用于长上下文大模型推理
06-18 EN

LUMEN: Load-Aware Coordinated Failure Recovery for Distributed LLM Serving
06-18 中

LUMEN：面向分布式大模型推理的负载感知协同故障恢复
06-14 EN

GF-DiT: Scheduling GPU Parallelism as a First-Class Resource for Diffusion Transformer Serving
06-14 中

GF-DiT：把 GPU 并行度当作可调度资源的扩散 Transformer 推理系统
06-07 EN

SlidingServe: SLO-Aware Sliding-Window Scheduling for LLM Inference
06-07 中

SlidingServe：面向LLM推理的SLO感知滑动窗口调度
05-28 EN

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
05-28 中

Mooncake：以 KV Cache 为核心的大模型推理服务解耦架构
05-21 EN

SGLang: Efficient Execution of Structured Language Model Programs — Technical Review
05-21 中

SGLang:为 LM 程序而生的前端 DSL + 协同设计运行时 —— 阅读笔记
05-20 EN

Sarathi-Serve: Taming the Throughput–Latency Tradeoff in LLM Inference — Technical Review
05-20 中

Sarathi-Serve:用 chunked-prefill 驯服 LLM 推理的吞吐-延迟权衡 —— 阅读笔记
05-17 EN

PipeSD: Cloud-Edge Collaborative Pipeline Inference with Speculative Decoding — Technical Review
05-17 中

PipeSD：基于推测解码的云边协同流水线推理框架 —— 阅读笔记
05-16 EN

An Interpretable Latency Model for Speculative Decoding in LLM Serving — Technical Review
05-16 中

用 Little 定律解释推测解码在真实服务中的提速曲线 —— 阅读笔记
05-12 EN

DAPO: An Open-Source LLM Reinforcement Learning System at Scale
05-12 中

DAPO：大规模开源 LLM 强化学习系统
05-10 EN

Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving
05-10 中

Tutti：让基于 SSD 的 KV Cache 真正适用于长上下文 LLM Serving
04-09 EN

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Deep Technical Review
04-09 中

DistServe：通过 Prefill/Decoding 解耦实现面向 Goodput 的大模型服务优化 — 深度阅读笔记
02-19 EN

vLLM and PagedAttention: Efficient Memory Management for Large Language Model Serving — Technical Review