A traffic jump rarely waits for a planned capacity upgrade. For a 15–30 person engineering team, I would investigate client retries before buying more compute: an overloaded API can turn one failed request into several, making its own recovery harder. The first scaling limit may be a contract between callers and the service, not the size of the service itself.
Retries can exhaust capacity before CPU does
What fails first when your simple API hits 10x traffic is a useful question, but the first failure worth fixing may sit outside the API process because callers decide how much work a slow response creates. A client that times out and immediately retries can increase incoming work while the original request is still running. CPU utilization may look manageable at that point because requests are waiting for a PostgreSQL connection, a lock, or another service rather than executing on a core.
Use a planning example to make the risk visible: at 120 original requests per second, if 20% of callers make two additional attempts after an initial failure, the service receives up to 168 attempts per second. Those are illustrative values, not a forecast; the calculation matters because a load test that counts only completed user operations would conceal the extra 48 attempts. If the retrying calls hold database connections while waiting, increasing the API replica count can add contenders without adding database capacity.
I would not start by raising every timeout and connection-pool limit, because that lets more requests wait longer while giving the constrained dependency no additional throughput. First, separate original requests from attempts in the metrics. Record the client attempt number, outcome, and elapsed time; record server-side active work and queue time independently. OpenTelemetry spans linked through the W3C Trace Context traceparent header can show whether several inbound attempts came from one user action, while Prometheus counters can quantify the amplification without requiring someone to inspect individual traces.
Check whether retries are even appropriate. An idempotent GET may be safe to retry, but a payment-like POST needs an idempotency key or an explicit no-retry policy because a timeout does not prove that the operation failed. For a proposed client policy, try a 3-second total deadline and at most one retry with jitter, then change those values against measured dependency latency and the user-facing latency objective. A total deadline is more useful than a per-attempt timeout alone because two full-length attempts can otherwise outlive the interaction the user has already abandoned.
Look at PostgreSQL’s pg_stat_activity for waiting sessions and enable pg_stat_statements to identify statements consuming time across calls. Pair those observations with p95 and p99 request latency rather than averages, because a small slow tail can trigger many synchronized retries even while the average looks healthy. The hypothesis is falsifiable: if attempts per original operation stay flat during a slowdown, retries are not your first scaling problem.
A closed-loop load test can hide the surge you need to survive
Case Study: Boosting App Performance with Load Testing can motivate a test, but a successful load-test result cannot establish that your own arrival pattern is safe because the generator may stop sending work when responses slow down. In a closed-loop test, each virtual user waits for a response before issuing the next request. As latency rises, the test produces fewer new requests—the opposite of a real surge in which independent users keep arriving.
For the first diagnostic run, use k6’s constant-arrival-rate executor and set its rate from a plausible peak, not from the largest number your laptop can produce. This runnable script sends GET requests to a service supplied through BASE_URL:
import http from 'k6/http';
export const options = {
scenarios: {
peak: { executor: 'constant-arrival-rate', rate: 120,
timeUnit: '1s', duration: '2m',
preAllocatedVUs: 100, maxVUs: 400 },
},
thresholds: { http_req_failed: ['rate<0.01'] },
};
export default function () {
http.get(`${__ENV.BASE_URL}/api/items`, { timeout: '3s' });
}
Run it with k6 run -e BASE_URL=http://localhost:8080 burst.js against a locally running service. The script’s 120 requests per second, two-minute duration, and 1% failure threshold are trial settings to replace with your traffic and reliability target; they are not universal pass marks. Watch k6’s dropped_iterations as well as http_req_failed, because a generator that cannot start scheduled iterations has not delivered the offered load. Watch http_req_duration alongside server-side latency, because a widening gap can locate waiting at a proxy or on the network.
Do not mistake a single endpoint for a production workload. Use access logs to estimate the mix of cheap reads, expensive reads, and writes, then preserve that mix when the test grows. Include the real client retry policy in a separate run; otherwise the baseline test answers “Can the server handle arrivals?” while leaving “What happens after failures begin?” unanswered. Keep test credentials and synthetic data isolated, because a realistic write mix should not create production side effects merely to improve the test’s fidelity.
Grafana dashboards should put offered arrivals, completed operations, retry attempts, queue depth, and dependency wait time on one time axis. That view exposes a useful sequence: arrivals stay constant, database waits rise, p99 latency crosses the client deadline, and attempts increase. Without the shared time axis, a team may attribute the last visible spike—often API CPU—to a problem that began at a dependency queue.
Admission control beats another replica when the dependency is full
Once requests are queuing, compare NGINX limit_req with application-level admission control rather than treating “rate limiting” as one decision. NGINX limit_req wins when a public endpoint needs a quick, coarse limit by client key: it can reject excess traffic before the application spends work parsing and dispatching it. Its cost is weaker knowledge of request expense and customer priority; a cheap lookup and a costly report may consume the same allowance.
Application-level admission control wins when the service can distinguish tenants, expensive operations, and work already in progress. A bounded semaphore around a costly database path, for example, protects that path even when several routes reach it. Its cost is code and operational ownership: every path that reaches the constrained resource must respect the bound, and the team must decide what callers receive when no slot is available. For a small engineering organisation, I would choose the application-level bound for a known expensive path and use a proxy limit only as a coarse outer guard, because the application knows which work actually consumes the scarce resource.
Make rejection cheaper and clearer than waiting indefinitely. HTTP 429 Too Many Requests and Retry-After give clients an explicit response to a rate limit; a short response is preferable to a connection held until its deadline when the service already knows it cannot admit the work. That only helps if clients honor the response, so exercise the actual SDKs and background workers under rejection. If every worker retries at the same interval, the limit can create a repeating burst; jitter and a shared retry budget reduce that synchronization.
Envoy circuit-breaker settings such as max_pending_requests offer another boundary for queued upstream work. They are useful when multiple services share a proxy configuration, but their cost is that a proxy cannot necessarily tell which application operation deserves a slot. Redis caching is a different lever: it wins for repeated reads with acceptable staleness, yet it costs invalidation logic and does little for write contention. I would not add Redis merely because a load test shows high latency, because a database lock or retry storm will remain after unrelated reads move into a cache.
The scaling fix fails if no one owns the caller contract
In a 15–30 person organisation, one team may own the API while several others own web clients, workers, and integrations. An API-side limit can be technically correct and still fail in production if callers interpret every non-2xx response as permission to retry. Give the retry contract an owner and a test: which methods may retry, which status codes qualify, how long the total deadline lasts, and how many attempts one operation may spend.
Keep the change narrow enough to verify. Before changing capacity, capture a baseline of original operations per second, attempts per operation, p99 latency, rejection rate, and PostgreSQL wait time during the same load window. Then change one boundary—such as the worker’s retry budget or the expensive path’s concurrency cap—and repeat the arrival-rate test. A lower p99 achieved only because many requests were rejected is not an improvement unless the rejection rate and caller behavior meet an agreed objective.
Put that objective in the release decision rather than leaving it on a dashboard. For example, a team might propose that a peak test sustain its chosen arrival rate while keeping attempts below 1.1 per original operation and reporting all deliberate rejections separately. The 1.1 figure is a candidate guardrail to validate against the product’s failure tolerance, not a standard. It forces a useful conversation: whether the team is willing to shed work promptly, or whether an apparently successful test is being propped up by callers doing hidden extra work.
The first useful action is to count attempts
Ask one engineer to add an original-operation ID and attempt number to the busiest caller this week. Run a short, constant-arrival-rate test against a safe environment and graph attempts per operation beside dependency wait time. If the lines rise together, cap retries before adding replicas. If they do not, the same measurement gives you permission to investigate the next constraint without guessing.