When an API slows under a traffic spike, adding a queue can look safer than keeping requests synchronous. I would default to bounded synchronous processing instead. A queue wins only when the product can accept a pending result and the team can operate retries, duplicates, and delayed work. For a DevOps engineer, that distinction determines which failures remain visible to callers and which become production work after the request ends.
The response contract should decide whether a queue belongs in the path
The choice is between bounded synchronous processing, where the request completes or receives a prompt rejection, and asynchronous queue-backed processing, where the API acknowledges work before it finishes. Bounded synchronous processing wins when callers need an answer immediately and the backend can meet its latency objective with controlled concurrency. Queue-backed processing wins when completion can wait and a pending state has a useful meaning to the caller. Its cost is a new operational contract: work can be delayed, retried, duplicated, or left for an operator to recover.
Ask what the caller would do with HTTP 202 Accepted. If it can store a job ID, poll a status endpoint, and tolerate completion later, a queue may absorb a short burst without tying up request connections. If it must decide whether an operation succeeded before continuing, returning 202 changes behavior rather than improving performance. I would not put every write behind a queue merely to flatten a traffic graph, because an acknowledgement is not proof that a worker completed the write.
A practical comparison is a checkout-like API action that reserves a scarce resource. Keeping the reservation synchronous lets the caller know whether it succeeded; the cost is capacity for peak requests and a deliberate rejection policy when that capacity runs out. Sending a confirmation email can be asynchronous because the reservation result does not depend on email delivery; the cost is monitoring how long confirmations wait and handling permanent delivery failures. The dividing line is the promise made by the response, not whether the handler happens to perform a write.
What fails first when your simple API hits 10x traffic helps identify the stressed component, but identifying a bottleneck does not establish that deferred completion is acceptable, because the caller’s contract remains unchanged. Likewise, Scalable Software Architecture Patterns for Modern Systems offers patterns to choose from, but a pattern should not outrank the observable behavior an API promises. Write down that behavior before selecting Redis, Amazon SQS, or another queue.
Bounded synchronous processing wins while overload can be rejected honestly
A synchronous API does not have to accept unlimited concurrent work. Put a finite admission limit ahead of the scarce dependency, such as a PostgreSQL connection pool or an upstream service. When the limit is reached, reject excess requests promptly with HTTP 429 Too Many Requests where the caller is rate-limited, or HTTP 503 Service Unavailable where capacity is temporarily unavailable. A Retry-After header can guide retries, but clients also need jitter because identical retry timers can create another spike. The operational goal is to preserve useful throughput rather than let waiting requests consume every connection.
Set the limit from measurements, not from the replica count alone. In an illustrative k6 run, suppose an endpoint sustains 120 requests per second at a measured p95 latency of 180 milliseconds, while p95 rises sharply beyond 150 requests per second. Those figures are example observations, not a capacity claim for your service. Use the bend in your own throughput-and-latency curve as a starting point, then inspect Prometheus metrics for CPU throttling, PostgreSQL active connections, pool wait time, and downstream errors. A low CPU reading does not justify more concurrency if requests are already waiting for database connections.
The following k6 script sends a constant arrival rate to a local endpoint. The rate of 200 requests per second, the two-minute duration, and the virtual-user limits are values to tune; the script’s purpose is to expose what happens when arrival rate exceeds comfortable capacity, not to prescribe a production load target.
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: { burst: { executor: 'constant-arrival-rate',
rate: 200, timeUnit: '1s', duration: '2m',
preAllocatedVUs: 100, maxVUs: 500 } },
};
export default function () {
const r = http.get(__ENV.URL || 'http://localhost:8080/items');
check(r, { 'expected status': x => [200, 429, 503].includes(x.status) });
}
Run it with k6 run load.js against an isolated environment. The check accepts overload responses so that their frequency stays visible rather than being mistaken for a script error; inspect the status-code breakdown separately. OpenTelemetry traces can show where admitted requests spend time, while NGINX limit_req can enforce a front-door rate policy. Neither replaces a concurrency bound at the application’s scarce dependency, because requests already admitted through NGINX can still pile up on the same pool.
Kubernetes Horizontal Pod Autoscaler with autoscaling/v2 may add replicas when its configured metric rises, but I would not rely on HPA as the only overload control because scaling reacts after demand appears and extra replicas can open more PostgreSQL connections. PgBouncer can reduce connection churn; it cannot make a slow query cheap. Keep a rejection path even if autoscaling usually catches up, and test that rejected requests do not perform partial writes.
A queue wins when delayed completion has a bounded, observable meaning
Choose asynchronous processing when a burst is temporary, each unit of work is independent enough to retry, and the caller can receive a status resource instead of a final result. HTTP 202 Accepted is appropriate for that acknowledgement, but the response should include a job identifier and a way to learn whether the work finished or failed. Without those, a queue moves uncertainty from the request path into a support ticket.
A queue is not extra processing capacity. If workers can complete 80 jobs per second and arrivals remain at 100 jobs per second, backlog grows by 20 jobs every second; that difference is arithmetic for a hypothetical workload, not a measured production rate. Increasing Amazon SQS retention or Redis memory only postpones the consequence. Define an age objective—for example, a proposed service target that 99% of accepted jobs begin within five minutes—and alert on oldest-message age as well as queue depth. Age tells an operator whether users are waiting too long; depth alone cannot, because job rates and runtimes vary.
Amazon SQS Standard queues use at-least-once delivery, so a worker must tolerate receiving the same message again. An idempotency key stored under a PostgreSQL unique constraint can prevent duplicate effects, provided the worker checks and records the result in the same transaction as the protected database change. Set the SQS visibility timeout longer than normal processing, then arrange extensions or safe retries for outliers; a timeout shorter than the work can make two workers process one job concurrently. Route repeatedly failing jobs to a dead-letter queue with an alert and a documented replay procedure, because a dead-letter queue without an owner is storage for unresolved incidents.
Publishing to a queue after committing a database change creates a gap: the process can crash between those actions. A transactional outbox closes that gap by recording the event in the same database transaction as the change, then publishing it separately. The outbox relay can publish more than once, so consumers still need idempotency. Kafka can serve workloads that need retained event logs and multiple consumers, but I would not adopt it for a single delayed task stream unless those requirements exist, because partitioning, consumer lag, and broker operation add failure modes the on-call team must understand.
Backlog also needs an admission policy. Decide whether the API stops accepting jobs when estimated completion time exceeds the published promise. Returning 202 indefinitely while queue age climbs is less honest than rejecting new work, because callers have no way to distinguish “pending” from “unlikely to finish soon.” KEDA can scale workers from queue metrics, but cap worker concurrency against PostgreSQL and upstream limits; otherwise a drained backlog can turn into a dependency outage.
The production test should compare outcomes, not peak request counts
Before choosing either approach, run the same burst and recovery profile against both designs. Record admitted requests, completed operations, p95 and p99 response latency, end-to-end completion time, database pool wait, and errors. For the queue design, also record oldest-message age, retry count, dead-letter count, and the time required to clear the backlog after arrivals fall. For the synchronous design, record the share of 429 and 503 responses and whether clients retry successfully. A higher count of accepted requests is not a win if many remain incomplete after the incident.
Use two failure drills as tie-breakers. First, make PostgreSQL unavailable briefly. A bounded synchronous API should fail promptly without accumulating a large in-process waiting room; a queued API should stop workers from cycling through rapid failures and should preserve jobs for later processing. Second, disable one downstream dependency while keeping the API healthy. Check that both designs expose the failure in alerts that page the right owner, because a green HTTP availability dashboard can conceal a stalled worker fleet.
Test client behavior as part of the system. For synchronous rejection, verify backoff with jitter and a retry budget so that clients do not amplify overload. For queued work, verify that a repeated submission with the same idempotency key returns the existing job rather than creating another one. Set the key’s retention to cover the period in which a client might legitimately retry; that period is a product decision supported by observed client behavior, not a queue default. Document what a caller sees when a job has permanently failed.
My decision rule is intentionally strict: keep the operation synchronous if the team cannot define a truthful pending state, even when a queue makes the request-latency chart look better. Choose the queue when delayed completion is part of the contract and the on-call rotation can demonstrate recovery from duplicate delivery, growing age, and failed jobs. Those tests make the operational cost explicit before it becomes an incident.
The first change should be a limit you can verify
Start by identifying the endpoint’s scarce dependency and measuring how many concurrent requests it can serve before pool wait or latency rises sharply. Add an admission limit below that point, then run a burst test that confirms excess work receives a prompt, retryable response without a partial write. If the required caller experience cannot tolerate that response, design a status-bearing asynchronous contract and test its backlog recovery before accepting the first job.