autoscaling-strategies

Choose the signal that tracks actual load: requests per second or in-flight concurrency for HTTP APIs, queue depth or backlog per consumer for queue workers, and CPU only for genuinely compute-bound batch work. Add a latency or error-rate guardrail that can trigger scale-out independently.

Flapping comes from symmetric scale-out and scale-in behavior in an under-damped control loop. Fix it with asymmetry: scale out quickly with a short stabilization window, scale in slowly with a long one, and set a cooldown at least as long as the time new capacity takes to become useful.

Use horizontal scaling whenever work can be spread across stateless instances, since it offers near-unbounded capacity and better failure isolation. Reserve vertical scaling for unsplittable stateful units like primary databases or in-memory indexes that must remain whole.

Keep a minimum replica floor or warm pool on latency-sensitive paths so scale-up adds to warm capacity. For serverless, use provisioned concurrency on user-facing routes, and gate readiness probes on actual cache and connection-pool warm-up completion.

Use predictive or scheduled scaling when demand follows a known periodic pattern, since it provisions before load arrives and hides cold-start latency. Always keep a reactive layer underneath, because a wrong forecast has no way to recover on its own.

CPU is the wrong signal for queue consumers because the bottleneck is usually downstream I/O, not compute. Scale on backlog per consumer or oldest-message age instead, which directly measures how far behind the fleet is.