Netflix Details its In-House LLM Serving Platform with Triton and vLLM

Netflix has developed an internal large language model (LLM) serving platform integrating Triton and vLLM to handle diverse model sizes and hardware, supporting real-time and batch workloads across CPUs and GPUs. The platform balances model management, scheduling, and custom model compatibility while maintaining consistent production workflows.