A product manager facing a tenfold traffic forecast is often asked to approve an “architecture upgrade” before anyone can name the constraint it will remove. I would scope the work in the opposite direction: find the first queue that grows, protect it, and fund a larger change only if the protected system still misses its target. Otherwise, a rewrite can consume the release window while the bottleneck survives.
The first scaling failure is often a queue the dashboard cannot see
At higher load, an API can fail before any component looks fully utilized because requests wait for a database connection, a worker slot, or a downstream response while aggregate CPU remains unremarkable. A product dashboard that shows only request rate and average latency will miss that wait, because averages conceal the slow requests and request rate says nothing about time spent queued.
Start with one user journey that matters commercially: for example, submitting an order and receiving confirmation. Trace it with OpenTelemetry, then put p95 and p99 latency, HTTP 5xx rates, and dependency wait time on a Grafana dashboard backed by Prometheus. The purpose is to locate where elapsed time accumulates, not to produce a prettier monitoring screen. If p99 rises while database execution time stays flat, waiting for a connection is a stronger hypothesis than a slow query; the distinction changes the ticket you write.
The symptom inventory in What fails first when your simple API hits 10x traffic is useful for an incident review, but I would not turn its ordering into a roadmap because several visible failures can share one exhausted pool. A timeout at the API edge, a failed retry, and a checkout error may all begin with the same connection wait. Count them as one capacity investigation until traces show otherwise.
Separate arrival rate from completion rate. In an illustrative planning exercise, assume the API receives 120 requests per second at the forecast peak but completes only 105. That 15-request-per-second gap is not spare work that the system can catch up on while arrivals continue; it is a growing backlog. Have engineering record when the gap begins, which dependency fills first, and how long recovery takes after the load falls. Those observations give the PM a scope boundary that “make the API scalable” does not.
Run the test against a representative journey and dataset, because a fast empty-database benchmark cannot reveal waits caused by real query plans or contention. Use k6 to ramp traffic rather than jumping straight to the forecast peak, and annotate deployments and configuration changes on the graphs so the team can identify the load level at which behavior changes. A test that merely proves the service can survive a short burst does not establish sustained capacity: queues may take time to fill, and retries may continue after the generator stops.
Replica counts are not a capacity plan until shared limits are counted
Adding API replicas can make a shared dependency fail sooner because each replica may open its own database pool. Suppose a proposed deployment has 12 replicas with a pool limit of 8 connections each: its potential demand is 96 connections before background jobs, administration, or other services are counted. PostgreSQL 16 documents a default max_connections of 100; that vendor-published default is not a safe application budget, because PostgreSQL also needs room for other clients and reserved connections. The exact production limit must be checked rather than inferred from the default.
Ask for a resource budget that names every consumer: API replicas, job workers, migrations, analytics queries, and operational access. Record the configured limits as well as observed peaks, because a pool that has not filled during a quiet test can still fill when every replica scales out. For PostgreSQL, this command groups current sessions by state and wait event; run it with a role permitted to inspect the activity you need:
psql "$DATABASE_URL" <<'SQL'
SELECT state, COALESCE(wait_event_type, 'none') AS wait_type,
COALESCE(wait_event, 'none') AS wait_event,
count(*) AS sessions
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY 1, 2, 3
ORDER BY sessions DESC;
SQL
The result is a snapshot, not proof of a root cause, because a short-lived queue can disappear between samples. Pair repeated pg_stat_activity observations with pool wait-time metrics and traces. If connection acquisition dominates, the first ticket might be to cap application pools, shorten unnecessarily long transactions, or evaluate PgBouncer; adding a Redis cache first would be a guess unless reads are demonstrably driving the wait.
Make the arithmetic part of the planning conversation. A value to tune might be a 60-second sustained-load window for detecting a growing queue; choose it against the journey’s normal burst pattern so a brief launch spike does not trigger unnecessary work. Another proposed target might be a 99.9% availability SLO for the confirmation endpoint, but the PM should agree what counts as a successful confirmation before accepting that number. A response that says “accepted” while the order is later lost is not success for this journey, regardless of the HTTP status.
This budget also exposes a common scoping error: counting server capacity without counting retry traffic. If clients retry a timeout immediately, the retry adds demand to a system that is already behind. Inspect client SDK settings, proxy policies, and worker behavior alongside the server, because reducing server-side latency will not fix a retry loop that multiplies requests during an outage.
Bounded admission is usually a better first purchase than a new architecture
Once the limiting resource is known, make overload behavior explicit. Envoy can enforce a request limit at the edge, and an API can return HTTP 429 with a Retry-After header in line with HTTP semantics in RFC 9110. That response is preferable to silently admitting work that will time out, because a caller can distinguish temporary rejection from an accepted operation. The limit still needs a product decision: which requests may wait, which may be rejected, and which must be protected.
Compare two named options for a saturated confirmation path. Bounded synchronous processing wins when users need an immediate, definitive result and the dependency can meet the target under a capped arrival rate; it costs some rejected peak requests and requires honest client-facing retry behavior. Amazon SQS-backed asynchronous processing wins when a delayed result is acceptable and bursts exceed the rate at which the dependency can safely work; it costs queue operations, worker ownership, status tracking, and a product state between “accepted” and “completed.” Moving work to SQS does not create database capacity, because workers will encounter the same limit if their concurrency is left unbounded.
I would not split the API into microservices merely to absorb this forecast, because a service boundary does not reduce the number of database connections or the cost of the underlying operation. I would also defer a cache until traces identify repeated, cacheable reads as a meaningful share of the constrained workload; otherwise, cache invalidation becomes new work without relieving the measured bottleneck.
If the team chooses asynchronous processing, ask about duplicate submissions before approving the estimate. A worker can process a message again after a failure, so a confirmation operation needs a durable idempotency key and an outcome that can be queried. If the team chooses bounded synchronous processing, ask how the web client handles a 429 and whether a user can safely retry. These are product acceptance criteria, not implementation details, because they determine whether the journey remains trustworthy under load.
Tickets should remove constraints in dependency order
An “upgrade the architecture” epic hides sequencing risk because it bundles measurement, overload policy, code changes, and rollout into one finish line. Use Scope an Architecture Upgrade Without Rewriting the Product to challenge the proposed change size, then require each ticket to name the constraint it should remove. A ticket without a predicted metric change is difficult to validate, because completing the code does not show whether capacity improved.
The first work package should establish a baseline: one representative k6 scenario, traces for the critical journey, pool-wait and queue-depth metrics, and a record of current limits. Its deliverable is a graph that identifies the knee in the load curve, not a dashboard count. The second package should bound demand at that knee, with a documented HTTP 429 policy or a worker-concurrency cap. Only then should a package change the constrained dependency, such as transaction duration, a specific query, or pooling configuration. This sequence keeps the intervention small enough to attribute an improvement to it.
Give each package an owner, a rollback condition, and a customer-visible acceptance test. If a pool change lowers wait time but increases database CPU enough to slow confirmation, the change has moved the constraint rather than removed it. If a request limit preserves p99 latency but rejects the most valuable traffic indiscriminately, it has protected the system at the wrong product cost. These outcomes should be visible in the scope document before implementation begins.
Reserve time for client behavior as well. A server-side limit is incomplete when a mobile or browser client immediately retries the same request, because the new traffic can defeat the limit. The PM does not need to choose Envoy flags or write the retry code, but does need to specify whether an accepted request may be repeated, how long a user should wait, and what status they see while work is pending. Those answers determine whether the estimate covers a server configuration change or a cross-client release.
The release gate should test overload, not merely normal operation
A successful low-load regression test is a weak gate for scaling work because the changed behavior is supposed to appear near saturation. Write the release gate before the fix: at the agreed forecast load, the critical journey must stay within its latency target, completion rate must keep pace with arrivals, and the constrained pool must stop accumulating waiters. Set a separately negotiated ceiling for rejected requests rather than hiding them inside a latency chart; fast 429 responses can make latency look excellent while customers fail to complete the journey.
Test recovery too. Run sustained load long enough to fill the relevant queue, reduce arrivals, and measure how long the system takes to return to normal. Check for duplicate confirmations, stuck messages, and delayed retries after the traffic generator stops, because those failures often appear after the graph’s apparent peak. Roll out with a small traffic share and a reversible limit so the team can respond if production client behavior differs from the test.
At the next planning meeting, ask engineering for one trace of the critical journey under rising load and a table of shared-resource limits. Use those artifacts to name the first growing queue and write one ticket that bounds it. Do not approve a broader migration until that ticket has a predicted metric change and an overload acceptance test; the forecast alone cannot tell you which architecture work is worth buying.