Lightbits set to release KV cache engine to boost GPU performance

Lightbits Labs has launched Inferra, a software engine that enhances AI inference by managing key-value cache across GPU memory tiers, boosting performance and reducing latency for large language models with long contexts. Inferra uses predictive prefetching to keep data near GPUs, supporting over 10 million token contexts and increasing concurrent session density significantly on standard hardware.