Making Knowledge Distillation Cheap Enough to Run at Scale

A new method for knowledge distillation in large language models reduces VRAM usage by caching teacher model outputs and using a memory-efficient KL-divergence loss, enabling cheaper and scalable training on a single GPU.