Together AI, Turbo & Kernel
May 2024 - PresentAdvisors: Ben Athiwaratkun (Senior Research Scientist, Together AI) · Shuaiwen Song (Vice President of Research, Together AI)
Together Post-Training
RL/SFT service with custom training loops, multi-LoRA and full-weight training.
Dissecting NVIDIA Blackwell and Blackwell Ultra: a multi-level microbenchmark study
Differentiable harness
Learning token-population geometry for sparse mixture-of-experts
QATFactory: a versatile, deployment-aligned framework for quantization-aware training and distillation of LLMs
QuantProxy: evaluating cheap proxies for quantized LLMs
Hadamard rotation for NVFP4 KV caches: no measurable gain and a zero-code penalty
When does a speech model know to hand off? Reading competence from hidden states for real-time escalation (NeurIPS 2026 Workshop RTCA)
ColdGPTQ: document-diverse calibration and count-adaptive damping for MXFP4 MoE quantization
On-policy residual learning for diffusion language models (ORL)
Motivation. Residual predictions change with the model during diffusion decoding, making stale offline residuals a poor match for the states encountered at inference.
Contributions.
- Co-advised ORL, combining fresh residual trajectories and entropy contraction with a co-designed training and inference system for diffusion language models.
Orthus + Loop model
FoldDecoding: accelerating decode attention over 2- and 4-bit KV caches through affine folding
Tail-likelihood reinforcement learning (TailRL)
Motivation. Expected-reward reinforcement learning can lose coverage over rare, high-reward rollouts, limiting both policy improvement and the gains available from additional inference-time samples.
Contributions.
- Developed tail-likelihood and finite-rollout advantage formulations that keep rare, high-reward outcomes influential during policy optimization.
- Built evaluation recipes comparing TailRL with expected-reward baselines across Pass@k, Best-of-k, and test-time scaling behavior.
Mix architecture speculator
Scaling law of speculators
LCFS
Motivation. Multi-tenant LLM serving needs hierarchical fairness and performance isolation across shared instances and clusters without sacrificing throughput.
Contributions.
- Contributed to design discussions around hierarchical fairness, vruntime-style accounting, and weight partitioning across distributed serving instances.
- Participated in experiments evaluating performance isolation and fairness under multi-tenant LLM serving workloads.
AgentGo
Motivation. Tool-using agents alternate between long-context reasoning and external actions, but request-centric runtimes either evict useful KV state too early or waste memory by pinning it too long.
Contributions.
- Co-developed the core idea of treating multi-turn agent workflows as first-class programs rather than isolated requests.
- Helped build the staged system path from telemetry and shadow prediction to offline replay, observability, and config-gated runtime integration for prediction-aware scheduling.
Taylor-Calibrate
Motivation. Converting pretrained Transformers into hybrid linear-attention models is brittle because copied projections do not initialize the new recurrent decay, write-gate, and output-gate dynamics.
Contributions.
- Led the Taylor-guided initialization method for converting pretrained Transformers into hybrid linear-attention students by calibrating value projections, recurrent timescales, write gates, and output gates.
- Formulated the two-stage calibration pipeline that combines Taylor-derived teacher attention statistics with brief layer-local alignment.
- Designed ablations to isolate initialization sensitivity across GDN gate choices and retained-layer policies.
- Evaluated conversion across model families, showing stronger zero-shot students and 4.9×-9.2× fewer training tokens to reach matched recovery targets.
- Released the open-source codebase that distills Qwen/Llama softmax-attention Transformers into hybrid linear-attention students built on GatedDeltaNet, using Taylor-series-informed initialization before staged distillation.
Agent Evolve
Motivation. Current LLM-based multiagent systems are largely static after deployment and lack mechanisms for continual adaptation across agents, skills, and populations.
Contributions.
- Built a bio-inspired LLM multiagent framework with pheromone-style memory, evolutionary division of labor, and skill inheritance for open-ended population adaptation.
- Studied population-level adaptation through competition, selection, and cross-generation strategy transfer.
OSCAR (NeurIPS 2026)
Motivation. INT2 KV-cache quantization offers large memory savings for long-context serving, but generic rotations are not aligned with downstream attention and naive INT2 often collapses model accuracy.
Contributions.
- Led the design of an offline spectral covariance-aware rotation method for 2-bit KV-cache quantization that calibrates attention-aware key/value rotations and clipping thresholds.
- Formulated attention-induced covariance targets for keys and values and helped develop the theoretical analysis for the frozen-error surrogate.
- Designed the mixed-precision serving policy that protects sink and recent tokens in BF16 while quantizing history tokens with fixed rotations.
- Helped shape the SGLang-compatible INT2 serving path with Triton kernels, reducing KV memory by about 8× and improving large-batch throughput by up to about 7× at near-BF16 accuracy.
- Drove open-source adoption: 500+ GitHub stars in the first month, llama.cpp support for local deployments, and community-driven vLLM integration bringing rotation-based KV-cache quantization to mainstream serving stacks.
- Paper at NeurIPS 2026; covered by Towards AI, MarkTechPost, ModelScope, QbitAI, and Synced with 100,000+ reads across platforms.
SmartKV
I-DLM (COLM 2026)
Motivation. Diffusion language models promise parallel generation but often lack introspective consistency and practical serving support, leaving a quality and throughput gap against autoregressive models.
Contributions.
- Helped build an inference path for introspective diffusion language models that reuses AR serving optimizations while supporting introspective strided decoding.
- Contributed to stationary-batch scheduling and serving evaluation, demonstrating about 3× higher throughput than prior diffusion LLM serving baselines while matching same-scale AR quality.
Squeeze Evolve (COLM 2026)
Motivation. Recursive self-aggregation improves reasoning quality, but uniform compute allocation across generation and aggregation wastes cost on easy subsets and under-allocates recovery on hard subsets.
Contributions.
- Helped develop a multi-model orchestration view of recursive self-aggregation, routing generation and aggregation between large and small models based on cross-model confidence.
- Owned coding-benchmark execution and evaluation pipelines, especially for LiveCodeBench V6, and supported ablations on routing thresholds and aggregation behavior across AIME 2025 and HMMT 2025.
- Demonstrated 30-40% compute reduction at matched accuracy or 5-7 point accuracy gains at equivalent compute.
- Paper accepted by COLM 2026.
SAW-INT4
Motivation. KV-cache memory is a key bottleneck in real LLM serving, but many compression methods break paged cache layouts, regular memory access, or fused attention kernels; we sought a simple INT4 design that works under deployment constraints.
Contributions.
- Helped develop a system-aware INT4 KV-cache quantization study for real LLM serving, identifying token-wise INT4 with block-diagonal Hadamard rotation as a practical accuracy-efficiency point.
- Supported fused rotation-quantization kernel integration for paged KV-cache serving, matching plain INT4 throughput while recovering most accuracy lost by naive INT4.
Osprey (EMNLP 2026)
Motivation. Drafters trained from scratch on target-specific traces have limited training budgets and transfer poorly across target models.
Contributions.
- Coauthored Osprey, which builds stronger, transferable drafters through target-agnostic pre-training and target-specific adaptation of pretrained small language models.
XoRL
Motivation. Building an RL and SFT training stack for coding and reasoning agents required more than model fine-tuning: it needed an end-to-end system that coupled sandboxed environments, distributed rollout workers, and multi-node training plus serving infrastructure while staying stable under long-context, MoE, and rapidly changing model variants.
Contributions.
- Built much of the training-side RL framework, including agent PPO trainers, asynchronous rollout and pipeline-training paths, and the execution flow that converts multi-turn agent-environment interaction into PPO and GRPO training batches.
- Owned the training pipeline that ingests rollout trajectories, computes advantage, and performs policy updates plus rollout-model weight synchronization for coding-agent post-training.
- Implemented asynchronous rollout, replay-queue mini-batching, and router-assisted batching between rollout and training workers to overlap trajectory generation with policy optimization.
- Developed trajectory/data transforms, token-level loss masks, stepwise-vs-trajectory advantage handling, rejection sampling, and batch balancing to improve GRPO signal quality and training stability.
- Scaled long-context training recipes to 16K-32K contexts using Ulysses sequence parallelism, remove-padding, chunked prefill, and per-GPU token-budget tuning for DeepCoder and DeepScaleR-style runs.
- Implemented sequence-parallel (SP) compatibility across the training stack so long-context post-training paths worked correctly with distributed attention, packed sequences, and rollout-to-training data flow.
- Built SP-compatible MoE-LoRA kernel paths to support efficient distributed post-training for expert models without breaking sequence-parallel execution.
- Integrated QuACK fused kernels into XoRL to improve kernel efficiency and support higher-throughput post-training recipes.
- Added Qwen3.5 support and completed model bring-up across configs, training paths, and distributed recipes for reliable experimentation.
- Diagnosed and fixed multi-node training failures (position_ids, cu_seqlens, attention-mask, and MoE dispatch issues) that destabilized distributed recipes across evolving model families.
CoderForge
Motivation. High-quality coding agents require strong trajectory data, stable post-training pipelines, and task-aligned optimization objectives for code generation.
Contributions.
- Led the training pipeline for OpenHands R2E-Gym & SWE-Bench-scale data: curated high-signal SWE-smith / Rebench examples and fixed attention-mask plus position-ID issues in XoRL.
- Distilled Qwen3-480B trajectories into a 30B coding model via supervised fine-tuning and activation distillation, then initiated MoE / RL scaling for Qwen3-30B to improve SWE-Bench solve rates.
- Designed per-token loss formulations for coding-trajectory distillation and model-quality improvement.
CREST (Turbo-reasoning)
Motivation. Reasoning models often under-think or over-think at test time, wasting tokens or missing correct solutions; we sought a training-free intervention that could be deployed in mainstream serving stacks.
Contributions.
- Co-developed the core idea of a training-free test-time steering method that identifies and modulates cognitive attention heads, improving accuracy by up to 17.5% and reducing token usage by 37.6% across reasoning benchmarks.
- Designed deployment paths for integrating CREST into vLLM and SGLang.
Ladder-Residual
Motivation. Large-model inference under tensor parallelism often suffers from communication stalls and weak overlap between communication and computation; we sought an architecture-runtime co-design that improves throughput without sacrificing model quality.
Contributions.
- Co-conceived the parallelism-aware residual design and helped shape the paper's system and evaluation story.
- Implemented and optimized the gpt-fast inference path with CUDA Graphs and PyTorch compile ("reduce-overhead") for large-model serving.
- Benchmarked performance across model scales (1B-405B) and TP world sizes (1, 2, 4, 8, 16), validating up to 30% end-to-end throughput improvement on 70B models with P2P enabled and up to 60% with P2P disabled.
CARE
Motivation. MLA-style attention can improve serving efficiency, but most pretrained checkpoints use GQA/MHA and cannot directly benefit; we sought a practical conversion path that preserves quality while lowering inference cost.
Contributions.
- Developed the core idea and empirical framing for upgrading pretrained attention into MLA-compatible forms.
- Proposed a conversion pipeline that upgrades pretrained attention (e.g. GQA) into multi-head latent attention (MLA) for faster inference without increasing KV-cache size.
- Ran the full experimental suite and carried out vLLM integration and theoretical analysis.
LEXICO
Pulsar / SGLang
Motivation. High-throughput serving requires lower KV overhead and more stable speculative decoding across cache-hit patterns, batch sizes, and multi-node deployments.
Contributions.
- Applied a Swift-KV caching strategy to accelerate prefill by reducing KV memory overhead and improving end-to-end latency.
- Designed and implemented KV-cache prompt caching for the Phoenix speculator in Pulsar, stabilizing acceptance rates and reducing end-to-end latency.
- Resolved tokenizer chat-template issues and Docker deployment bugs for reliable multi-node operation, then benchmarked cache behavior across batch sizes and cache-hit scenarios to explain acceptance-rate variability and optimize cache-hit logic.
- Integrated and implemented Llama 4 support for sliding window attention.
Axolotl
Contributions.
- Integrated long-context attention (Ulysses, Ring Attention) into Axolotl and supported SFT data flow from successful trajectories to extend supervised post-training to larger context windows.
