provider-architecture

What problem does it solve?

This skill addresses the challenge of integrating 66+ LLM providers behind a single, trait-object based LLMProvider interface, enabling scalable routing, consistent error handling, and maintainable architecture in Rust gateways.

Core Features & Use Cases

• Unified provider abstraction for plug-and-play LLMs, including OpenAI-compatible APIs and alternative providers.
• High-performance gateway routing with a single connection pool, deterministic request handling, and centralized cost/health management.
• Use Case: Deploy a gateway that routes user requests to multiple providers (OpenAI, Azure, Google Vertex) based on model name, with fallbacks and cost-aware routing.

Quick Start

Start by implementing the LLMProvider trait for a provider, configure via environment variables, then initialize the provider and register it with the gateway router. Use a single GlobalPoolManager for HTTP calls and expose model information through the static registry. Run the gateway with the provider configuration loaded from env vars (e.g., OPENAI_API_KEY, AZURE_OPENAI_API_KEY, etc.).