llm-quota-orchestration

What problem does it solve?

Prevents quota exhaustion and cache disruption in multi-key LLM deployments by managing API key rotation, maintaining session affinity, and orchestrating tiered fallbacks so requests remain fast and reliable.

Core Features & Use Cases

• Sticky Sessions & Prefix Caching: Pin workflows to a single API key to preserve prefix cache hits and reduce latency.
• Tiered Resilience: Failover from Gemini cloud keys to local vLLM instances and provide a controlled fail-fast option when necessary.
• Quota-aware Key Selection & Ledger: Select keys by affinity, remaining quota, or LRU, and update a ledger using rate-limit headers with cooldown management.
• Use Case: Ideal for services with parallel LLM requests that must avoid 429s, maintain prompt caching benefits, and route traffic across multiple Gemini API keys.

Quick Start

Use the llm-quota-orchestration to validate available Gemini keys, pin a session to the best key, and fallback to a local model on quota exhaustion.