Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens

Shopify developed Gisting, a method that compresses large LLM prompts into smaller learned tokens, reducing inference latency and costs without changing model weights. This technique cut a 6000-token prompt to 1500 gist tokens, improving throughput and lowering GPU needs.