/ZHONG-JOO JOH/
"Let everything happen to you. Beauty and terror. Just keep going. No feeling is final." — Rainer Maria Rilke
I am a Senior Research Scientist at the Turbo Team, Together AI, supervised by Ben Athiwaratkun.
I am a Ph.D. at the School of Computer Science, Faculty of Engineering, The University of Sydney, supervised by Prof. Shuaiwen Leon Song. I have been fortunate to intern at Dolby, DeepSpeed Microsoft, Weixin Group Tencent, and Microsoft (China), contributing to projects in building machine learning systems. I was also a research associate at the School of Computer Science and Engineering, Sun Yat-sen University from 2019 to 2022, under the supervision of Prof. Dan Huang and Yutong Lu. I received my B.E. degree from the School of Computer Science and Engineering, Sun Yat-sen University in 2019.
My research spans efficient machine learning and systems, from model pretraining quality to efficient algorithms and system co-design that bridges emerging ML/LLM methods and real-world applications, improving both productivity (usable, robust stacks) and performance (throughput, memory, cost-efficiency). I work across academia and industry (Together AI), with an emphasis on LLM efficiency and scalable training infrastructure. My research is supported by Together AI.
Feel free to drop me an email if you have aligned interests.
(🔥 indicates the projects I am leading)
For the full categorized list across all five research themes, see the Research page.
Quantization · 2026
Attention-aware offline rotations + clipping that compress the long-history KV cache to 2.28 bits per element on Qwen3-4B/8B/32B and GLM-4.7-FP8 — ~8× KV-memory reduction and up to ~7× higher large-batch throughput, with mean accuracy gap to BF16 of just 1.42 / -0.02 / +0.27 on the 8B / 32B / 358B models. SGLang INT2-serving path with Triton kernels.
Efficient inference
First diffusion language model to match same-scale AR quality. Introspective Strided Decoding (ISD) verifies prior tokens while advancing new ones in one forward pass. I-DLM-8B beats LLaDA-2.1-mini (16B) by +26 on AIME-24 and +15 on LiveCodeBench-v6 with half the parameters, at 2.9-4.1× throughput. With gated LoRA, R-ISD is bit-for-bit lossless versus the base AR model.
Speculator training system · ICML 2026
Unified training-serving system that closes the loop by continuously learning a speculator from live inference traces. Day-0 deployment, 1.45× speedup on MiniMax M2.1 and Qwen3-Coder-Next, and 1.25× over static speculators.
Efficient inference · ICLR 2026
Covariance-aware low-rank decomposition that converts pretrained GQA/MHA into MLA — up to 215× lower one-shot perplexity and 1.70× mean accuracy at matched KV budgets on Llama-3.1-8B/70B and Qwen3-4B/30B.
Quantization · MLSys 2026
Dynamic Channel-wise Precision Boost keeps a small fraction of K-cache channels at 4-bit while quantizing the rest to 2-bit — cuts KV memory ~8×, enables up to 8× larger batches, and achieves 2.1×–4.1× higher throughput at matched memory.
Efficient inference · ICML 2025
Architectural modification that decouples communication from computation in tensor parallelism — 29% end-to-end wall-clock speedup on a 70B Transformer with TP=8. Pure PyTorch, no custom CUDA kernels.
Efficient training · NeurIPS 2024
Task-aware adapters built via SVD on covariance-oriented decomposition. Two modes: knowledge-preserved (mitigates forgetting, beats LoRA) and instruction-previewed (matches full fine-tuning, beats LoRA/DoRA/PiSSA).
Quantization · USENIX ATC 2024
TC-FPx — first full-stack GPU kernel scheme with unified Tensor-Core support for FP6 and arbitrary bit-widths. Delivers 1.69×–2.65× higher inference throughput over FP16 and enables LLaMA-70B on a single GPU.
Quantization · VLDB 2024
Load-as-Sparse and Compute-as-Dense (LSCD) methodology for unstructured-sparse SpMM on GPU tensor cores — 2.9× faster than Sputnik at the kernel level and up to 3.8× over DeepSpeed end-to-end on OPT models.
RL training system · DeepSpeed
End-to-end RLHF training pipeline replicating InstructGPT. Trains a 6.7B model in ~4.1 hours / ≈$132 on 8×A100; a 66B model in 2.1 days / ≈$1,620.
Software engineering · ICSE 2020
Hybrid plagiarism-detection framework for JavaScript mini-games. Uses edit-distance + network-flow on V8/Ignition bytecode plus a priority-queue framework to consolidate detection algorithms. Deployed at Tencent.
See the full archive for every announcement.
Hybrid & San Francisco, United States
Research Consultant / Senior Research Scientist May 2024 - Present
Sydney, Australia
Research Intern Mar 2024 - Sep 2024
Sydney, Australia & Remote
Research Intern Mar 2023 - Feb 2024
Champaign, IL, US & Guangzhou, China
Research Intern Jul 2018 - Jul 2020
Guangzhou, China
Project Assistant to Senior Cloud Architect Sep 2018 - Feb 2019
Sydney, Australia
Doctor of Philosophy (Ph.D.) Oct 2022 - Feb 2026
Sydney, Australia
Visiting Scholar Mar 2022 - Oct 2022
Guangzhou, China
Research Associate Sep 2019 - Mar 2022
Remote & Champaign, IL, US
Summer Session Student Jun 2018 - Sep 2018
Guangzhou, China
Bachelor of Engineering in Computer Science and Technology Sep 2015 - Jun 2019
Selected accepted conference and venue-published papers. For the complete list — including journals, preprints, under-review submissions, in-preparation work, books, and patents — see the Publications page or my Google Scholar profile.
Efficient inference through covariance-aware low-rank decomposition.
COMP3520: Operating Systems Internals — Tutor, The University of Sydney, Fall 2023.
Programming Languages: Pascal (11 yrs), C (11 yrs), C++ (11 yrs), Python (6 yrs), HTML, CSS, JavaScript (6 yrs), Java (6 yrs), SQL (6 yrs), Bash (6 yrs), LaTeX (5 yrs), Matlab (5 yrs), CUDA (5 yrs), R (4 yrs), Go (4 yrs), Triton (2 yrs)
Systems and Infrastructure: MPI/OpenMPI (6 yrs), Linux Kernel (5 yrs), Distributed/Parallel File Systems — Lustre, HDFS (5 yrs), Kubernetes, Kubernetes Scheduler, Kubernetes SR-IOV (5 yrs), Docker (5 yrs), Hadoop (4 yrs), Spark (4 yrs), YARN (4 yrs), Mesos (4 yrs), NVLink (2 yrs), NVSHMEM (2 yrs), TensorCore, CudaCore Programming (2 yrs)
Machine Learning and AI: TensorFlow (5 yrs), PyTorch (4 yrs), TorchServe (4 yrs), TensorBoard (4 yrs), Ray (4 yrs), JAX (2 yrs), Triton (2 yrs), DeepSpeed (1 yr), HuggingFace (1 yr), Reinforcement Learning (4 yrs), CNN, RNN, ResNet, Attention Block, UNet, Transformer, ViT (5 yrs), Neural Architecture Search (3 yrs), Diffusion Models (1 yr), GPT-2/3/4 (1 yr), RLHF (1 yr), VeOmni (1 yr), TorchTitan (1 yr), FSDP (1 yr), ZeRO 1/2/3 (1 yr)
Databases and Storage: MySQL (6 yrs), Oracle SQL (6 yrs), MongoDB (6 yrs), PostgreSQL (6 yrs), Redis (4 yrs), Hive SQL (3 yrs)
Front-end Development: PHP (6 yrs), Vue.js (6 yrs), ReactJS (6 yrs), ASP.NET (6 yrs), jQuery (6 yrs), AngularJS (6 yrs), Apache (6 yrs), MeteorJS (6 yrs)
Back-end Development: Spring Boot (6 yrs), Django (6 yrs), Flask (6 yrs), Node.js (6 yrs), Express (6 yrs), REST API Design (6 yrs), CI/CD
Mobile Development: Android Studio (6 yrs, Java/Kotlin), XCode (6 yrs, Swift/Objective-C), React Native (6 yrs, cross-platform), Flutter (6 yrs, cross-platform)
Web Crawling & Testing: Urllib (6 yrs), BeautifulSoup (6 yrs), Scrapy (6 yrs), Requests (6 yrs), JSON (6 yrs), Selenium (6 yrs), Pytest (6 yrs), JUnit (6 yrs)
Version Control & Build Systems: Git (8 yrs), Gradle (6 yrs, Android/Java), Maven (6 yrs, Java), npm (6 yrs, JavaScript), pip (6 yrs, Python)
Development Tools & Libraries: Airflow (2 yrs), Kafka (2 yrs), Elasticsearch (2 yrs), OpenCV (5 yrs), Pandas (5 yrs), NumPy (5 yrs), SciPy (5 yrs), NLTK (5 yrs), Matplotlib (5 yrs), Seaborn (5 yrs), Azure Data Factory (2 yrs), AWS (2 yrs), Google Cloud Platform (2 yrs)
Fitness: Fencing (6 yrs), Jogging (7 yrs), Bodybuilding (6 yrs, Hongxing Fitness Club Outstanding Students), Table Tennis (11 yrs), Badminton (11 yrs).
Leisure: Web & mobile application development (5 yrs), Saxophone (9 yrs), Magic (1 yr), Video Games (500+ PS5 game collections).
HPCA 2026 Conference Volunteer (Registration & On-site Operations) Feb 2026
SYSU School of Computer Science and Engineering, Student Union — Vice President Jul 2016 - Jul 2017
Changjun High School Volunteer Jul 2013 - Jul 2014