329 posts in total. Keep on posting.

2026

• 04-02 中

  GPipe：微批次流水线并行的大规模模型训练 — 深度阅读笔记

  GPipe 提出了微批次流水线并行方法，实现大规模神经网络的高效训练。本文从零讲解流水线调度算法、梯度累积、重计算内存优化，以及在 AmoebaNet 和 Transformer 上的实验结果。

• 04-02 EN

  GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism — In-Depth Technical Review

  GPipe introduces micro-batch pipeline parallelism for efficient training of large neural networks. This review covers the pipeline scheduling algorithm, gradient accumulation, re-materialization for memory optimization, and experimental results on AmoebaNet and Transformer models.

• 04-01 EN

  Layer Pruning for Efficient Large Language Models — In-Depth Technical Review

  Layer pruning removes redundant layers from LLMs to reduce compute and memory costs. Covers layer importance metrics, pruning strategies, and fine-tuning recovery.

• 03-31 EN

  Constitutional AI: Harmlessness from AI Feedback — In-Depth Technical Review

  Constitutional AI trains harmless AI assistants using AI-generated feedback instead of human labels. Covers the critique-revision pipeline, RLAIF, and comparison with RLHF.

• 03-30 EN

  Chain-of-Thought Prompting Elicits Reasoning in LLMs — In-Depth Technical Review

  Chain-of-Thought prompting enables LLMs to perform complex reasoning by generating intermediate steps. Covers few-shot CoT, zero-shot CoT, and analysis across arithmetic, commonsense, and symbolic tasks.

• 03-29 EN

  Ring Attention: Blockwise Transformers for Near-Infinite Context — In-Depth Technical Review

  Ring Attention enables near-infinite context length by distributing attention computation across devices in a ring topology. Covers blockwise computation, online softmax, and memory analysis.

• 03-28 EN

  Mamba: Linear-Time Sequence Modeling with Selective State Spaces — In-Depth Technical Review

  Mamba introduces selective state space models as an alternative to Transformers with linear-time complexity. Covers selective scan, hardware-aware algorithms, and language modeling results.

• 03-27 EN

  GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection — In-Depth Technical Review

  GaLore reduces memory requirements for LLM training through gradient low-rank projection. Covers the mathematical foundation, subspace switching, and memory savings analysis.

• 03-26 EN

  Alpa: Automating Inter- and Intra-Operator Parallelism — In-Depth Technical Review

  Alpa automates the search for optimal parallelism strategies combining data, tensor, and pipeline parallelism. Covers the ILP formulation, inter-operator DP, and compilation framework.

• 03-25 EN

  GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — In-Depth Technical Review

  GPTQ enables efficient post-training quantization of large language models to 3-4 bits with minimal accuracy loss. Covers the layer-wise quantization algorithm, Hessian-based error correction, and practical deployment.

• 03-24 EN

  Proximal Policy Optimization Algorithms — In-Depth Technical Review

  PPO is one of the most influential RL algorithms. This review covers policy gradients, TRPO, the clipped surrogate objective, and PPO's role in RLHF/LLM alignment.

• 03-24 中

  近端策略优化算法（PPO）— 深度阅读笔记

  PPO（近端策略优化）是深度学习时代最具影响力的强化学习算法之一。本文从零开始详细讲解策略梯度、TRPO 到 PPO 裁剪目标的完整推导，覆盖 MuJoCo、Atari 实验分析，以及 PPO 在 RLHF/LLM 对齐中的核心作用。