azure-aigateway

Create an APIM backend pointing to your Azure OpenAI endpoint, enable managed identity on APIM, grant it the Cognitive Services User role, and import the OpenAI API specification. Then apply governance policies like token limits and semantic caching in the inbound section.

Add azure-openai-semantic-cache-lookup in the inbound policy and azure-openai-semantic-cache-store in outbound, with an embeddings backend and Azure Cache for Redis Enterprise with RediSearch. Use a score-threshold around 0.8; note it does not work with streaming responses.

No. Semantic caching and token metrics policies are not compatible with streaming responses using "stream": true. Use non-streaming requests for cost control scenarios that rely on caching.

A 401 usually means managed identity is not enabled on APIM, the Cognitive Services User role is missing, or the authentication resource is wrong. Verify identity with az apim show, check role assignments, and wait 5-10 minutes for RBAC propagation.

Increase the tokens-per-minute value, load balance across multiple Azure OpenAI backends, or enable semantic caching to reduce token consumption. If the 429 comes from Azure OpenAI itself, raise the deployment TPM quota or add retry with backoff.

Yes. Use the rate-limit-by-key policy with a counter-key derived from an agent identifier header such as X-Agent-Id. This throttles calls per agent and can return remaining-calls and Retry-After headers.