Error rate in performance testing

Run the load at rising concurrency and divide non-2xx by total at each level. npx autocannon@8 -c 20 -d 5 -j on the shedding route below returned 144 722 non-2xx out of 152 666, a 94.8% error rate, where 8 connections returned zero. One number without a level says nothing.

Why check this

An error rate is the figure a release gate is written against, and the figure most often quoted with no load attached. Run this before any change to capacity, a pool size, or a rate limiter.

The failure it prevents is an error budget computed from the wrong counter. A run in which every connection was refused reports zero non-2xx, because no response arrived to carry a status. Divide non-2xx by total there and the service scores 100% healthy.

Generator and target share this machine's eight cores, so every count describes the pair.

Prerequisites

• Node 22 or later, and npx autocannon@8, which fetches autocannon on first use.
• A free port. Port 9671 is used below, and port 9679 is left with nothing on it on purpose.
• The target, saved as errors-server.mjs. /shed returns 503 above 8 requests in flight, /limited returns 429 above 400 requests a second, /leak returns 503 once 3 000 leases are gone, /slow answers after 3 s, and /stats reports what the server itself counted, per second.

// errors-server.mjs
import { createServer } from 'node:http';
let t0 = Date.now(), inflight = 0, leased = 0, tokens = 400, log = [];
setInterval(() => { tokens = 400; }, 1000).unref();
const mark = (k) => { const s = Math.floor((Date.now() - t0) / 1000); (log[s] ??= { ok: 0, err: 0 })[k] += 1; };
const send = (res, code, body) => { res.writeHead(code, { 'content-type': 'application/json' }); res.end(body); };
createServer((req, res) => {
  const u = req.url;
  if (u === '/stats') return send(res, 200, JSON.stringify({ leased, log: [...log].map((b, s) => ({ s, ok: b?.ok ?? 0, err: b?.err ?? 0 })) }));
  if (u === '/reset') { t0 = Date.now(); inflight = 0; leased = 0; log = []; return send(res, 200, '{"reset":true}'); }
  if (u === '/slow') return void setTimeout(() => send(res, 200, '{"ok":true}'), 3000);
  if (u === '/limited') {
    if (tokens-- <= 0) { mark('err'); return send(res, 429, '{"error":"rate limited"}'); }
    mark('ok'); return send(res, 200, '{"ok":true}');
  }
  if (u === '/leak') {
    if (++leased > 3000) { mark('err'); return send(res, 503, '{"error":"pool exhausted"}'); }
    mark('ok'); return void setTimeout(() => send(res, 200, '{"ok":true}'), 5);
  }
  if (++inflight > 8) { inflight -= 1; mark('err'); return send(res, 503, '{"error":"shedding"}'); }
  mark('ok');
  setTimeout(() => { inflight -= 1; send(res, 200, '{"ok":true}'); }, 5);
}).listen(9671, '127.0.0.1', () => console.log('listening on 9671'));

• Send load only to a host you own.

Steps

1. 1

   Step 1.

   Confirm the port has no listener.

   netstat -ano | grep ":9671 " | grep -c LISTENING
   

   0

2. 2

   Step 2.

   Start the target.

   node errors-server.mjs > errors-server.log 2>&1 & sleep 2; netstat -ano | grep ":9671 " | grep LISTENING
   

     TCP    127.0.0.1:9671         0.0.0.0:0              LISTENING       16316

3. 3

   Step 3.

   Measure the error rate as a curve, one run per connection count.

   curl -s http://127.0.0.1:9671/reset > /dev/null && for c in 2 8 9 10 12 20 50; do npx autocannon@8 -c $c -d 5 -j http://127.0.0.1:9671/shed > c$c.json; node -e "const r=require('./c$c.json'),p=(x,n)=>String(x).padStart(n);const t=r['2xx']+r.non2xx;console.log('-c'+p($c,3)+'  total'+p(t,8)+'  2xx'+p(r['2xx'],7)+'  non2xx'+p(r.non2xx,8)+'  rate'+p((100*r.non2xx/t).toFixed(1)+'%',7))"; done
   

   -c  2  total     646  2xx    646  non2xx       0  rate   0.0%
   -c  8  total    2560  2xx   2560  non2xx       0  rate   0.0%
   -c  9  total  100236  2xx   7991  non2xx   92245  rate  92.0%
   -c 10  total  152675  2xx   7984  non2xx  144691  rate  94.8%
   -c 12  total  159062  2xx   7984  non2xx  151078  rate  95.0%
   -c 20  total  152666  2xx   7944  non2xx  144722  rate  94.8%
   -c 50  total  156028  2xx   7339  non2xx  148689  rate  95.3%

   The rate is a cliff, not a slope. One extra connection, from 8 to 9, takes it from 0.0% to 92.0%.

   Now read the 2xx column. It holds near 7 980 from 9 connections to 20 while the rate climbs from 92.0% to 95.3%, so the count of callers served barely moved.

   That is the generator's loop, not the server's health. A refusal costs nothing, so a refused connection asks again at once while a served one waits. Quote the served count beside the rate.

4. 4

   Step 4.

   Separate the four states a single error counter collapses.

   for case in "5xx|-c 20 -d 3|http://127.0.0.1:9671/shed" "429|-c 20 -d 3|http://127.0.0.1:9671/limited" "timeout|-c 5 -d 3 -t 1|http://127.0.0.1:9671/slow" "refused|-c 5 -d 3|http://127.0.0.1:9679/"; do IFS='|' read -r name flags url <<< "$case"; npx autocannon@8 $flags -j "$url" > e.json 2>/dev/null; node -e "const r=require('./e.json'),p=(x,n)=>String(x).padStart(n);console.log('$name'.padEnd(8)+' sent'+p(r.requests.sent,7)+'  completed'+p(r.requests.total,7)+'  errors'+p(r.errors,6)+'  timeouts'+p(r.timeouts,4)+'  non2xx'+p(r.non2xx,7)+'  codes '+JSON.stringify(r.statusCodeStats))"; done
   

   5xx      sent 115193  completed 115173  errors     0  timeouts   0  non2xx 110452  codes {"200":{"count":4721},"503":{"count":110452}}
   429      sent 106943  completed 106923  errors     0  timeouts   0  non2xx 105323  codes {"200":{"count":1600},"429":{"count":105323}}
   timeout  sent     15  completed      0  errors    10  timeouts  10  non2xx      0  codes {}
   refused  sent  21195  completed      0  errors 21190  timeouts   0  non2xx      0  codes {}

   Four server states, four signatures. A 503 and a 429 differ in no column except statusCodeStats, and the summary collapses both into one non-2xx total. They are opposite findings: shedding means capacity ran out, a 429 means the service declined on purpose.

   A timeout raises errors and timeouts equally and leaves non2xx at zero, so timeouts is a subset of errors. A refused connection raises errors alone. Both leave statusCodeStats empty.

   The refused row is the one to remember: 21 195 sent, 0 completed, non2xx 0 and 2xx 0. A rate written as non-2xx over 2xx plus non-2xx divides zero by zero.

5. 5

   Step 5.

   Read where in the run the errors fell, and compare the two counts.

   curl -s http://127.0.0.1:9671/reset > /dev/null && npx autocannon@8 -c 10 -d 10 -j http://127.0.0.1:9671/leak > leak.json && curl -s http://127.0.0.1:9671/stats > stats.json && node -e "const g=require('./leak.json'),s=require('./stats.json');for(const b of s.log)console.log('second '+String(b.s).padStart(2)+'   ok '+String(b.ok).padStart(6)+'   err '+String(b.err).padStart(7));const so=s.log.reduce((a,b)=>a+b.ok,0),se=s.log.reduce((a,b)=>a+b.err,0);console.log('server   ok '+so+'   err '+se);console.log('client  2xx '+g['2xx']+'   non2xx '+g.non2xx+'   errors '+g.errors)"
   

   second  0   ok      0   err       0
   second  1   ok    500   err       0
   second  2   ok    870   err       0
   second  3   ok    780   err       0
   second  4   ok    750   err       0
   second  5   ok    100   err   34560
   second  6   ok      0   err   42164
   second  7   ok      0   err   42824
   second  8   ok      0   err   42913
   second  9   ok      0   err   41917
   second 10   ok      0   err   43065
   second 11   ok      0   err   12608
   server   ok 3000   err 260051
   client  2xx 3000   non2xx 260041   errors 0

   The run-level rate is 98.9% and it describes nothing that happened. The service was perfect for five seconds, then it never served another request. A depleting resource has that shape: errors begin at a request count, not at a rate, so it fails after enough traffic at any speed.

   The two counts differ by 10, the connection count, because the run ended with 10 responses in flight. Believe the server for what it did and the client for what a caller received. Step 4 settles it: 21 190 failures the server never saw.

6. 6

   Step 6.

   Run a route that fails at a rate rather than after a budget, and compare the shape.

   curl -s http://127.0.0.1:9671/reset > /dev/null && npx autocannon@8 -c 10 -d 10 -j http://127.0.0.1:9671/limited > lim.json && curl -s http://127.0.0.1:9671/stats > stats2.json && node -e "const g=require('./lim.json'),s=require('./stats2.json');for(const b of s.log)console.log('second '+String(b.s).padStart(2)+'   ok '+String(b.ok).padStart(6)+'   err '+String(b.err).padStart(7));console.log('client  2xx '+g['2xx']+'   non2xx '+g.non2xx+'   errors '+g.errors)"
   

   second  0   ok      0   err       0
   second  1   ok    800   err   25132
   second  2   ok    400   err   42513
   second  3   ok    400   err   42731
   second  4   ok    400   err   39913
   second  5   ok    400   err   37778
   second  6   ok    400   err   35237
   second  7   ok    400   err   41837
   second  8   ok    400   err   36761
   second  9   ok    400   err   30223
   second 10   ok    400   err   39915
   second 11   ok    400   err   40627
   second 12   ok      0   err   10066
   client  2xx 4800   non2xx 422723   errors 0

   Errors in the first second and every second after, with ok pinned at exactly 400, the configured cap. A flat error line beside a flat success line is a limiter working; the fix is a client that backs off.

   Both runs report about 99% errors. One was healthy for five seconds then dead, the other served its quota throughout. Only the per-second view separates them.

7. 7

   Step 7.

   Stop the target and confirm the port is clear.

   netstat -ano | grep "127.0.0.1:9671 " | grep LISTENING; powershell -Command "Stop-Process -Id 16316 -Force"; sleep 1; netstat -ano | grep ":9671 " | grep -c LISTENING
   

     TCP    127.0.0.1:9671         0.0.0.0:0              LISTENING       16316
   0

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | non2xx high, errors 0 | The service answered every request, with a failure status | Read statusCodeStats before deciding whether it shed load or declined it. | | 429 in statusCodeStats | A limiter refused on purpose | Check the limit and the client's backoff, not the server's capacity. | | 503 in statusCodeStats | The service ran out of something | Find the resource, then rerun below the level where it ran out. | | errors equal to timeouts | Requests were abandoned by the generator | The server may still be working on them. Read its own counters. | | errors high, statusCodeStats empty | Connections never completed | Check that the port is right and the listener is up before reading anything else. | | Errors only in the last seconds | Something depletes with traffic | Rerun longer. The rate is a function of total requests, not of load. | | Errors in every second, ok flat | A cap that is working | Report the cap, not the percentage. |

Common mistakes

What to check next

• How to read load test results: what a non-zero error count does to every other cell.
• How to test graceful degradation: whether 503 with Retry-After is the intended answer.
• How to test rate limiter under burst: the 429 case, including refill after the window.
• How to stress test an api: the level where the first non-2xx appears.
• How to test API error responses: whether the failure bodies are usable.

FAQ

What is an acceptable error rate in performance testing?

The service sets it, next to the load it holds at and the codes it counts. One percentage with no concurrency and no code list passes both runs in step 4.

How do I evaluate error rate in api testing?

Take the level, the 2xx count, the count per status code, and the timeout and connection error counts. Divide failures by attempts, not by responses, then look at when they happened.

Do timeouts count as errors?

In autocannon they are counted twice: timeouts is a subset of errors, and the run above reported 10 of each. The deadline is the generator's, set with -t, so raising it moves the count without changing the server.

Verified