resilience-patterns

Track consecutive failures and open the circuit after a threshold, rejecting calls until a recovery timeout elapses, then test with a half-open state. The Skill provides an async CircuitBreaker class with CLOSED, OPEN, and HALF_OPEN states wrapping any awaitable call.

Use a decorator that catches exceptions, waits base_delay times 2 to the attempt power plus random jitter, and re-raises after the final attempt. The Skill's retry decorator caps delay with a max_delay parameter to prevent excessive waits.

A bulkhead limits concurrent access to a resource using a semaphore, isolating failures so one slow dependency cannot exhaust all capacity. A circuit breaker stops calling a failing service entirely until it recovers, preventing wasted requests.

Never retry non-transient errors like HTTP 400 responses, which will fail identically on every attempt. Also avoid retrying without backoff, since simultaneous retries create a thundering herd that overwhelms recovering services.

Use idempotency keys: store each processed request keyed by a unique idempotency key and return the existing result when the same key arrives again. This makes retried operations safe without charging customers twice.