prompt-caching

Anthropic prompt caching stores repeated prompt prefixes so subsequent requests reuse the cached prefix instead of reprocessing it. Structure prompts with stable content first and variable content last, and verify current provider documentation for minimum cacheable sizes and billing.

Cache augmented generation pre-loads documents directly into the cached prompt instead of retrieving them at query time like RAG. CAG works well when the document set fits in the context window and changes infrequently, eliminating retrieval latency.

Response caching stores full LLM outputs keyed by the query, returning the cached result for identical or semantically similar requests. Semantic similarity matching often matters more than exact match, and cached entries need invalidation rules to avoid stale answers.

Prompt caching fails when the prefix changes between requests, since caches match on exact prefixes. Keep static content like system instructions at the start of the prompt and place dynamic user content at the end.

Avoid caching responses generated with high temperature, since outputs vary intentionally, and avoid caching time-sensitive answers without invalidation. Caching everything indiscriminately leads to stale or incorrect responses over time.