Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models

Researchers analyzed and compared the performance of Diffusion Language Models (DLMs) and Autoregressive Language Models (ARMs) in natural language processing tasks. They found that while DLMs offer higher parallelism and arithmetic intensity, they struggle with scaling to longer contexts, unlike ARMs. Techniques like block-wise decoding and reducing sampling steps could improve DLM efficiency.