Training LLMs using Off-Policy vs On-Policy Distillation
Most practitioners distill by training on teacher outputs — that’s off-policy distillation, and it has a fundamental flaw. On-policy distillation fixes it at...
Recent Posts
Scaling LLM Inference with Disaggregated Prefill and Decode
Disaggregated prefill/decode isn’t an academic concept it’s the fix that saved my LLM serving at scale for 40M users. Here’s exactly how it works and why mon...
Deep Dive into Fine Tuning a LoRA Reranker on Phi-3
I fine tuned Phi-3 as a pairwise reranker with LoRA and logged every gradient. Early layers changed 200x more than late layers, but ranking representations o...
Text Diffusion: The Idea Hiding Inside BERT All Along
Auto regressive generation is sequential and diffusion uses much fewer passes in text generation.
Cross Entropy Loss Connection to GPT Models
Cross-entropy loss isn’t a heuristic, it is maximum likelihood estimation with a sign flip. It also shows how the same math powers GPT training.