cost-aware-llm-pipeline

What problem does it solve?

LLM-based applications often incur high costs due to using the highest-performance models for all tasks, repeated system prompts, and untracked retries. This pattern provides a cost-aware approach to model routing, immutable cost tracking, controlled retries, and prompt caching to keep budgets in check.

Core Features & Use Cases

• Model Routing: automatically select the most cost-effective model based on task complexity.
• Immutable Cost Tracking: maintain a ledger of costs without mutating state.
• Narrow Retry Logic: retry only transient errors with exponential backoff to avoid wasted spend.
• Prompt Caching: cache system prompts to prevent unnecessary token usage on repeated runs.

Quick Start

Run the pipeline on sample input to observe automatic model routing, cost tracking, retries, and prompt caching in action.