Engineering Shortcuts Inflate Your On-Call Bill

A passing load test can still authorize a release that takes production down. The problem is often not insufficient traffic in the test; it is that nobody agreed what the…

A passing load test can still authorize a release that takes production down. The problem is often not insufficient traffic in the test; it is that nobody agreed what the result permits the team to ship. I would treat load testing as a release admission decision, with explicit overload behavior and rollback conditions, rather than as a contest to find the highest requests-per-second number.

A green percentile is not evidence that a release is safe

A single p95 latency threshold is attractive because it gives CI a clean pass or fail, but it can hide a dangerous result when failed requests disappear from the latency story. A service that quickly returns HTTP 503 may look faster under pressure while serving fewer users. Check success rate, completed throughput, latency, and rejected work together; each describes a different outcome of the same test.

Your Load Tests Break First as Traffic and Teams Scale identifies a real hazard, but I would not make the test harness the default suspect after a surprising pass: the more costly mistake is accepting a result without proving what traffic the harness actually delivered. In k6, a constant-arrival-rate executor schedules new iterations independently of response time, while dropped_iterations records arrivals it could not start. If arrivals drop, a latency pass describes the requests that ran, not the demand the team meant to test.

The distinction is practical rather than academic. Consider a proposed target of 40 arrivals per second for 2 minutes: that schedules 4,800 arrivals if the generator keeps up. If it starts only 3,600, the missing 1,200 are a test-capacity failure, not proof that the API handled the target. Conversely, 4,800 started iterations do not prove 4,800 useful responses; HTTP status and application-level checks still matter. Those figures are an example calculation, not a benchmark for any particular service.

This k6 script makes generator drops, HTTP failures, and latency independent gates. The rates and thresholds are values to tune against the service’s own objective; run it with k6 run -e API_URL=http://127.0.0.1:8080/api/items scenario.js against an endpoint you control.

import http from 'k6/http';
import { check } from 'k6';
export const options = {
  scenarios: { arrivals: {
    executor: 'constant-arrival-rate', rate: 40, timeUnit: '1s',
    duration: '2m', preAllocatedVUs: 20, maxVUs: 100
  } },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<300'],
    dropped_iterations: ['count==0']
  }
};
export default function () {
  const r = http.get(__ENV.API_URL);
  check(r, { 'expected status': x => x.status === 200 });
}

Keep the status check even though k6 also reports HTTP failures: a 200 response containing an application error can pass a transport-level test. Where that response shape exists, add a body assertion before using this script as a release gate. The operational cost of skipping that step is a CI pass that on-call cannot reconcile with customer reports.

A traffic model that omits rejection makes overload look like a mystery

Teams often replay a steady request mix because it is easy to reproduce, then meet a production burst they never specified. A fixed average conceals whether clients retry together, whether expensive endpoints dominate the burst, and whether each rejected request creates more work. Record the traffic contract alongside the test: arrival pattern, endpoint mix, payload sizes, client timeouts, retry policy, and the response expected after capacity is exhausted. Without those fields, a green result is difficult to compare with an incident.

Closed-loop VUs and open-loop arrivals answer different questions. Closed-loop virtual users, available in tools such as k6 and JMeter, win when the question is how a fixed population experiences a workflow: each user waits for a response before continuing. Their cost is coordinated omission under overload, because slow responses naturally reduce the offered request rate. Open-loop arrivals, such as k6’s constant-arrival-rate, win when the question is whether the service can sustain a specified external arrival rate. Their cost is that the load generator must have enough workers and network capacity; otherwise its drops can be mistaken for server behavior.

That choice determines what an on-call engineer can infer. At a hypothetical measured mean response time of 1 second, sustaining 40 completed requests per second implies roughly 40 requests in flight by Little’s Law. If latency grows to 3 seconds at the same arrival rate, concurrency tends toward 120 unless something rejects or sheds work. A test capped at 100 k6 VUs will then start dropping iterations, so the team must either provision more generator capacity or report that the intended load was not delivered. The numbers illustrate a queueing relationship, not a promised capacity.

Rejection should be a designed response, not an accidental timeout. For an HTTP API, decide whether an exhausted admission limit returns 429 or 503, whether Retry-After is appropriate, and whether callers use exponential backoff with jitter. A proxy limit such as NGINX limit_req can protect an upstream before its worker queue grows; it also costs legitimate callers requests if the limit ignores differing endpoint costs. Test both the accepted requests and the rejected ones, because a clean 429 is preferable to an accepted request that holds a connection until the client gives up.

What fails first when your simple API hits 10x traffic is a useful failure inventory, but an inventory cannot establish whether an overload response is acceptable for a particular release. The production decision needs a stated ceiling and a client-visible behavior at that ceiling.

Autoscaling the API can spend the database’s remaining capacity

A common response to a failing API test is to raise the Kubernetes HorizontalPodAutoscaler maximum, because more pods can reduce CPU pressure in the application tier. That change can make the incident worse when each new pod opens database connections and the database is already the bottleneck. Before changing maxReplicas in an HPA autoscaling/v2 manifest, calculate the worst-case connection demand from replica count, per-pod pool size, background jobs, and administrative connections.

Suppose a proposed configuration permits 20 replicas with a pool of 30 connections each. Its potential application demand is 600 connections before workers and operator sessions are counted. Those are configuration values, not a claim about PostgreSQL’s universal limit. Compare that demand with the actual max_connections setting and inspect pg_stat_activity during the test. If the database cannot serve the extra concurrent queries, adding pods creates more contention rather than more completed work.

PgBouncer can reduce connection churn, but I would not add it merely to make a connection graph look calmer: transaction pooling can break code that assumes session state persists, so the application’s use of prepared statements and session settings must be checked first. A smaller application pool plus an admission limit may be the safer change when query concurrency, rather than connection setup, is the constraint. That option costs some requests an explicit rejection; unchecked scaling costs accepted requests unpredictable waits and may also delay recovery.

Correlate the layers during a load run instead of declaring a winner from API CPU alone. Prometheus can show completed request rate, error rate, and a latency histogram such as http_request_duration_seconds_bucket; PostgreSQL can show active versus waiting sessions; OpenTelemetry traces can show whether time moved into a database span. Check Envoy’s upstream pending and overflow metrics if Envoy is in the path, because a proxy queue can absorb demand long enough to conceal an exhausted service. Metric names depend on instrumentation, so verify the deployed exporters before turning a dashboard query into a gate.

The cost of missing this dependency budget is operational: the team discovers the true limit during the rollout, then has to distinguish a bad application build from a database queue while customers are waiting. A pre-release test that records both tiers gives on-call a narrower question to answer.

A load test without a rollback decision only documents the risk

Teams sometimes attach graphs to a change request and call that performance approval. I would not approve a release on graphs alone, because graphs leave the person on call to invent a decision rule while the rollout is already increasing traffic. Write the rule before the run: which failure is a blocker, which degradation is tolerated, which production signal stops the rollout, and who can reverse it.

For example, a team might set a candidate gate of less than 1% HTTP failures and no generator drops for a defined workload, then use a separate production SLO and error-budget policy to govern rollout. The gate is a proposed threshold, not a universal reliability target. Its purpose is to prevent an obviously unsuitable build from advancing; it cannot certify production because staging data, caches, and competing workloads differ. If the SLO is measured with Prometheus histograms, define the request population and error classification before comparing test and production figures, or the two percentages may have different denominators.

Run the same request mix against the candidate and the current release, then preserve the k6 output, deployment revision, dataset description, and relevant PostgreSQL and proxy snapshots. A comparison against the current release is more informative than a standalone “pass” when the environment has changed, because both builds face the same test conditions. It still costs test time and infrastructure, so reserve the full comparison for changes that affect request paths, concurrency, dependencies, or admission behavior rather than running it indiscriminately on every documentation edit.

During a canary, make rollback executable rather than aspirational. Name the deployment revision to restore, the traffic shift to undo, and the signals that trigger the action. If an HTTP 503 spike appears only after increasing canary traffic, the operator should be able to stop the shift without debating whether a staging p95 once passed. That clarity is what turns a load test from a performance report into production support.

Start with the last load test that passed: find its scheduled arrivals, started iterations, completed responses, and failure classifications. If any one is missing, rerun a small controlled workload with those figures recorded before raising the next release threshold. That first check will tell you whether the existing “pass” describes delivered traffic, served traffic, or merely an unloaded generator.