Spike testing in software testing

Raise the offered rate in one step, hold it for seconds, drop it back, and keep watching. node spike.mjs http://127.0.0.1:9762 step 0 took a service from 100 to 900 requests per second for three seconds. Nothing succeeded from second 5, and the queue emptied ten seconds after the spike ended.

Why check this

Run this before a launch that sends traffic at a known minute: a campaign, a store opening, a batch job that wakes every client at once. Repeat it after any change to a pool size, a queue, a cache or a retry policy.

The failure it catches is a service that survives the spike and stays down after it. Below, the offered rate returns to normal at second 7 and the first success arrives at second 17, because the queue built during the spike is still draining.

Two limits. One machine runs both generator and target, so these are not capacity figures. A spike over loopback has no connection setup in it, the part of a real spike that costs most. And nothing here can show autoscaling, which needs a second machine and a scheduler.

Prerequisites

• Node 22 or later. No load tool is used: the shape of the offered rate is the instrument.
• A free port. Port 9762 is used below. netstat -ano | grep ":9762 " | grep -c LISTENING prints 0 when free.
• The target, saved as spike-server.mjs and started with node spike-server.mjs &. Eight workers at 25 ms each: a steady capacity of 320 per second.

// spike-server.mjs  -  a pool of 8 workers, 25 ms each, and a queue in front of them
import { createServer } from 'node:http';
const POOL = 8, WORK_MS = 25;                      // steady capacity: 8 / 0.025 = 320 per second
let inflight = 0, completed = 0;
const queue = [];
function pump() {
  while (inflight < POOL && queue.length > 0) {
    inflight += 1;
    const done = queue.shift();
    setTimeout(() => { inflight -= 1; completed += 1; done(); pump(); }, WORK_MS);
  }
}
createServer((req, res) => {
  if (req.url === '/stats') {
    res.writeHead(200, { 'content-type': 'application/json' });
    return res.end(JSON.stringify({ inflight, depth: queue.length, completed }));
  }
  queue.push(() => { res.writeHead(200, { 'content-type': 'application/json' }); res.end('{"ok":true}'); });
  pump();
}).listen(9762, '127.0.0.1', () => console.log('spike server on 9762'));

• spike.mjs, the generator. It offers a rate rather than holding connections open, so the load does not shrink when the service slows, and it watches for 28 seconds while sending for 12.

// spike.mjs <base> <step|ramp> <retries>  -  offers a shaped load and reports each second
const [base, shape, retries] = [process.argv[2], process.argv[3], +process.argv[4]];
const BASE = 100, PEAK = 900, SPIKE = [4, 7], DEADLINE = 1000, TRAFFIC = 12, WATCH = 28;
const rateAt = (t) => (t < SPIKE[0] || t >= SPIKE[1] ? BASE
  : shape === 'step' ? PEAK
  : BASE + ((PEAK - BASE) * (t - SPIKE[0])) / (SPIKE[1] - SPIKE[0]));
const t0 = Date.now();
const sec = () => Math.floor((Date.now() - t0) / 1000);
const rows = [];
const row = (s) => (rows[s] ??= { rate: 0, sent: 0, ok: 0, timeout: 0, retry: 0, ms: 0, depth: 0 });
async function send(isRetry) {
  const s = sec(); if (s >= WATCH) return;
  const k = row(s); k.sent += 1; if (isRetry) k.retry += 1;
  const started = performance.now();
  try {
    const r = await fetch(`${base}/api`, { signal: AbortSignal.timeout(DEADLINE) });
    await r.arrayBuffer();
    k.ok += 1; k.ms += performance.now() - started;
  } catch {
    k.timeout += 1;
    if (retries && !isRetry) setTimeout(() => send(true), 200);   // one retry, never two
  }
}
let owed = 0, last = Date.now();                   // scale by measured elapsed time, not by 20
const tick = setInterval(() => {
  const now = Date.now(), t = (now - t0) / 1000;
  if (t >= TRAFFIC) return clearInterval(tick);
  owed += (rateAt(t) * (now - last)) / 1000;
  last = now;
  row(sec()).rate = Math.round(rateAt(t));
  while (owed >= 1) { owed -= 1; send(false); }
}, 20);
const poll = setInterval(async () => {
  const s = sec(); if (s >= WATCH) return clearInterval(poll);
  if (!row(s).rate) row(s).rate = Math.round(rateAt(s));
  try { row(s).depth = (await (await fetch(`${base}/stats`)).json()).depth; } catch { row(s).depth = -1; }
}, 1000);
await new Promise((r) => setTimeout(r, WATCH * 1000 + 200));
let drained = -1;
for (let s = TRAFFIC; s < WATCH; s += 1) if ((rows[s] ?? row(s)).depth === 0) { drained = s; break; }
const stop = drained === -1 ? WATCH - 1 : drained;
console.log(' s   rate   sent     ok  timeout  retry   mean ms   depth');
for (let s = 0; s <= stop; s += 1) {
  const k = rows[s] ?? row(s);
  const mean = k.ok ? (k.ms / k.ok).toFixed(0) : '-';
  console.log(`${String(s).padStart(2)}  ${String(k.rate).padStart(5)}  ${String(k.sent).padStart(5)}  ${String(k.ok).padStart(5)}  ${String(k.timeout).padStart(7)}  ${String(k.retry).padStart(5)}  ${String(mean).padStart(8)}  ${String(k.depth).padStart(6)}`);
}
console.log(drained === -1
  ? `queue still holding ${(rows[stop] ?? row(stop)).depth} at ${stop}s, ${stop - SPIKE[1]}s after the spike ended`
  : `queue empty at ${drained}s, ${drained - SPIKE[1]}s after the spike ended`);
const t = rows.reduce((a, k) => ({ sent: a.sent + k.sent, ok: a.ok + k.ok, to: a.to + k.timeout, re: a.re + k.retry }), { sent: 0, ok: 0, to: 0, re: 0 });
console.log(`totals  sent ${t.sent}  ok ${t.ok}  timeout ${t.to}  of which retries ${t.re}`);

Steps

1. 1

   Step 1.

   Send the step change: 100 per second, 900 for three seconds, then 100 again.

   node spike.mjs http://127.0.0.1:9762 step 0
   

    s   rate   sent     ok  timeout  retry   mean ms   depth
   0    100     98     98        0      0        44       0
   1    100    100    100        0      0        45       0
   2    100     99     99        0      0        44       0
   3    100    100    100        0      0        46       0
   4    900    907    340      567      0       514      23
   5    900    906      0      906      0         -     671
   6    900    896      0      896      0         -    1308
   7    100    101      0      101      0         -    1927
   8    100     99      0       99      0         -    1773
   9    100    101      0      101      0         -    1616
   10    100     97      0       97      0         -    1461
   11    100    101      0      101      0         -    1301
   12    100      0      0        0      0         -    1144
   13    100      0      0        0      0         -     880
   14    100      0      0        0      0         -     624
   15    100      0      0        0      0         -     368
   16    100      0      0        0      0         -     112
   17    100      0      0        0      0         -       0
   queue empty at 17s, 10s after the spike ended
   totals  sent 3605  ok 737  timeout 2868  of which retries 0

   Read sent first: it tracks rate to within 11 requests, so the generator kept its promise and the failures belong to the server. Second 3 is healthy at 46 ms, second 4 has 567 timeouts, and there is no warning second. Now the right half: the offered rate is back to 100 at second 7 and nothing succeeds until 17.

2. 2

   Step 2.

   Send the same peak as a ramp over the same three seconds.

   node spike.mjs http://127.0.0.1:9762 ramp 0
   

    s   rate   sent     ok  timeout  retry   mean ms   depth
   0    100     97     97        0      0        42       0
   1    100    100    100        0      0        44       0
   2    100    100    100        0      0        44       0
   3    100    101    101        0      0        42       0
   4    364    235    235        0      0        54       0
   5    629    498    460       38      0       522      21
   6    900    771      0      771      0         -     283
   7    100    108      0      108      0         -     778
   8    100    100      0      100      0         -     626
   9    100     99      0       99      0         -     477
   10    100    100     46       54      0       868     322
   11    100    102    102        0      0       416     166
   12    100      0      0        0      0         -       0
   queue empty at 12s, 5s after the spike ended
   totals  sent 2411  ok 1241  timeout 1170  of which retries 0

   Same peak, different failure. Second 5 is the warning the step never gave: 460 answers at a mean of 522 ms with 38 timeouts, a service bending before it breaks. The ramp also offers less traffic, 2411 against 3605, which is part of what a ramp is. It served 1241 against 737 and recovered at second 11, while traffic was still arriving.

3. 3

   Step 3.

   Repeat the step with a client that retries once, 200 ms after a timeout.

   node spike.mjs http://127.0.0.1:9762 step 1
   

    s   rate   sent     ok  timeout  retry   mean ms   depth
   0    100     97     97        0      0        37       0
   1    100    101    101        0      0        43       0
   2    100    101    101        0      0        45       0
   3    100     99     99        0      0        45       0
   4    900    893    357      536      0       516      21
   5    900   1247      0     1247    334         -     664
   6    900   1797      0     1797    905         -    1700
   7    100   1011      0     1011    910         -    3149
   8    100    374      0      374    273         -    3897
   9    100    197      0      197     99         -    3913
   10    100    201      0      201     99         -    3858
   11    100    201      0      201    101         -    3798
   12    100    100      0      100    100         -    3736
   13    100     22      0       22     22         -    3579
   14    100      0      0        0      0         -    3334
   15    100      0      0        0      0         -    3070
   16    100      0      0        0      0         -    2814
   17    100      0      0        0      0         -    2558
   18    100      0      0        0      0         -    2302
   19    100      0      0        0      0         -    2046
   20    100      0      0        0      0         -    1790
   21    100      0      0        0      0         -    1542
   22    100      0      0        0      0         -    1294
   23    100      0      0        0      0         -    1038
   24    100      0      0        0      0         -     790
   25    100      0      0        0      0         -     541
   26    100      0      0        0      0         -     286
   27    100      0      0        0      0         -      30
   queue still holding 30 at 27s, 20s after the spike ended
   totals  sent 6441  ok 755  timeout 5686  of which retries 2843

   The offered rate is unchanged and the second spike is the clients' own. The peak second went from 907 requests to 1797, and second 7, where the rate is back to 100, carried 1011. Successes did not move: 755 against 737. The backlog peaked at 3913 against 1927, and 20 seconds after the spike the queue still held 30 where the run without retries was empty at 10.

4. 4

   Step 4.

   Count what the generator itself spent.

   netsh int ipv4 show dynamicport tcp | tail -4; echo "sockets in TIME_WAIT: $(netstat -ano | grep -c TIME_WAIT)"
   

   ---------------------------------
   Start Port      : 49152
   Number of Ports : 16384
   
   sockets in TIME_WAIT: 7599

   One 15 second run left 7599 sockets in TIME_WAIT against a pool of 16384. A timed-out request abandons its connection, so every failure costs a port reserved for minutes. Run the test twice without waiting and the generator, not the service, is what fails.

5. 5

   Step 5.

   Stop the target and confirm the port is clear.

   netstat -ano | grep "127.0.0.1:9762 " | grep LISTENING; powershell -Command "Stop-Process -Id 39956 -Force"; netstat -ano | grep ":9762 " | grep -c LISTENING
   

     TCP    127.0.0.1:9762         0.0.0.0:0              LISTENING       39956
   0

   Read the process id from the last column and stop it.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Failures in the second the rate jumps | Nothing absorbed the step | Compare step with ramp before blaming capacity. | | One slow second, then failures | A ramp warned where the step did not | Test the shape your real traffic has. | | Failures continue after the rate returns to normal | Recovery, not the spike | Report time to first success after the spike. | | Queue depth falling after all traffic stops | Work is queued for callers that left | Bound the queue and drop work past its deadline. | | sent above rate | Retries are adding load | Cap retries, add jitter, and stop retrying on a timeout. | | sent below rate | The generator fell behind | Fix the generator before reading the table. | | connect EADDRINUSE from the generator | Its ephemeral ports are exhausted | Wait for TIME_WAIT to clear, or generate from two hosts. | | Recovery longer than the spike | The queue sets it, not the handler | Size the queue from the recovery you can accept. |

Common mistakes

What to check next

• Load test ramp up time: why a run's first seconds read differently.
• How to test graceful degradation: what turns this failure into a smaller one.
• How to stress test an api: the steady limit this spike went past.
• Error rate in performance testing: counting failures in the recovery window.
• How to check connection limit of a server: the far end of the ports step 4 counted.

FAQ

What is the difference between spike testing and stress testing?

Stress testing raises load until the service breaks and records where. Spike testing jumps once inside a fixed shape and records what the jump did, including after it ends. The number here was not a breaking point but ten seconds of recovery from three seconds of load.

How long should a spike last?

Long enough to build the queue, and far shorter than the window you watch. Three seconds of spike needed 17 seconds of observation here, and 27 was not enough once clients retried.

Can I test autoscaling this way?

Not on one machine. Scaling needs a scheduler, more than one instance and a health signal, none of which exist here. What this test gives a scaling policy is the number it must beat: one instance's recovery time, measured from the end of the spike.

Why does the service keep failing after the spike ends?

Because the work did not stop when the traffic did. In step 1 the queue held 1927 requests at second 7 and drained at 320 per second, so the service spent ten seconds answering callers who had gone. An exhausted pool or a retry storm in flight does the same.

Verified