How to check trace propagation between services

Send one request through the chain with a known traceparent, then compare the log lines each service wrote. The trace id must be identical at every hop and the parent id must change at every hop. A service that forwards the header unchanged leaves two spans claiming the same parent.

Why check this

Propagation is the half of trace context that goes untested, because the header looks right on the wire at every hop. Two services can pass a valid traceparent between them and still produce a trace no backend can draw, and the symptom is a missing service rather than a bad header.

Run this after a change to the HTTP client a service uses, after a framework or middleware upgrade, and whenever a new service is inserted into an existing path. The failure it prevents: the order service forwards the header it received instead of minting its own span id, so the payment service it calls is drawn as its sibling rather than its child. An engineer reading that trace concludes order never called payments.

The chain below runs on one machine, so none of this measures latency or network behaviour. It measures what each service does to the header, which is the part that breaks.

Prerequisites

• Node 22 and no packages. Output below came from v22.23.2.
• curl. Any build reads and sends these headers; the output below came from 8.21.0.
• The W3C Trace Context recommendation, section 3.4, which states that a vendor must set the parent id to the id of the current operation on every outgoing request.
• Three chained services, A calling B calling C, each logging what it received and what it sent. Save as chain.mjs and start with node chain.mjs > chain.log 2>&1 &. Setting FORWARD_UNCHANGED=1 makes service B forward the header it received, which is the defect step 4 reproduces.

import { createServer } from 'node:http';
import { randomBytes } from 'node:crypto';

const PORT = { A: 19731, B: 19732, C: 19733 };
const TP = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;

function parseTraceparent(h) {
  const m = typeof h === 'string' && TP.exec(h);
  if (!m) return null;
  if (/^0+$/.test(m[1]) || /^0+$/.test(m[2])) return null;
  return { traceId: m[1], parentId: m[2], flags: m[3] };
}

function mutateTracestate(raw, key, value) {
  const kept = (raw ?? '').split(',').map((s) => s.trim()).filter(Boolean)
    .filter((e) => e.split('=')[0] !== key);
  const list = [`${key}=${value}`, ...kept];
  return { header: list.slice(0, 32).join(','), dropped: list.slice(32) };
}

function service(name, port, next) {
  const key = 'svc' + name.toLowerCase();
  createServer(async (req, res) => {
    const tpIn = req.headers.traceparent ?? null;
    const tsIn = req.headers.tracestate ?? null;
    const ctx = parseTraceparent(tpIn) ?? { traceId: randomBytes(16).toString('hex'), parentId: null, flags: '01', restarted: true };
    const span = randomBytes(8).toString('hex');
    const ts = mutateTracestate(ctx.restarted ? null : tsIn, key, span);
    let tpOut = null;
    const body = { svc: name };
    if (next) {
      tpOut = process.env.FORWARD_UNCHANGED === '1' && name === 'B' ? tpIn : `00-${ctx.traceId}-${span}-${ctx.flags}`;
      const r = await fetch(`http://127.0.0.1:${next}/`, { headers: { traceparent: tpOut, tracestate: ts.header } });
      body.downstream = await r.json();
    }
    console.log(JSON.stringify({ kind: 'tp', svc: name, in: tpIn, traceId: ctx.traceId, parentSpan: ctx.parentId, span, restarted: Boolean(ctx.restarted), out: tpOut }));
    console.log(JSON.stringify({ kind: 'ts', svc: name, in: tsIn, out: ts.header, dropped: ts.dropped }));
    res.setHeader('content-type', 'application/json');
    res.end(JSON.stringify(body));
  }).listen(port, '127.0.0.1');
}

service('C', PORT.C, null);
service('B', PORT.B, PORT.C);
service('A', PORT.A, PORT.B);
console.log(`A ${PORT.A} -> B ${PORT.B} -> C ${PORT.C}`);

• Stop the chain when you finish. Find its process id with netstat -ano | grep 19731 and stop that one id, not every node process on the machine.

Steps

1. 1

   Step 1.

   Send one request into the front of the chain with a trace id you will recognise.

   curl -s -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' http://127.0.0.1:19731/
   

   {"svc":"A","downstream":{"svc":"B","downstream":{"svc":"C"}}}

   All three services answered, so the request reached the end of the chain. That is the part a status code can tell you, and it is the last thing it can tell you.

2. 2

   Step 2.

   Read the traceparent line each service logged.

   grep '"tp"' chain.log
   

   {"kind":"tp","svc":"C","in":"00-4bf92f3577b34da6a3ce929d0e0e4736-23da5af9de835a5e-01","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","parentSpan":"23da5af9de835a5e","span":"5daecad284d465a0","restarted":false,"out":null}
   {"kind":"tp","svc":"B","in":"00-4bf92f3577b34da6a3ce929d0e0e4736-15b335cfa1b1e607-01","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","parentSpan":"15b335cfa1b1e607","span":"23da5af9de835a5e","restarted":false,"out":"00-4bf92f3577b34da6a3ce929d0e0e4736-23da5af9de835a5e-01"}
   {"kind":"tp","svc":"A","in":"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","parentSpan":"00f067aa0ba902b7","span":"15b335cfa1b1e607","restarted":false,"out":"00-4bf92f3577b34da6a3ce929d0e0e4736-15b335cfa1b1e607-01"}

   C logs last because it answers first: the chain unwinds from the far end.

3. 3

   Step 3.

   Reduce the three lines to the identifiers that decide the verdict.

   grep '"tp"' chain.log | sed -E 's/.*"svc":"([A-C])".*"traceId":"([0-9a-f]+)","parentSpan":("[0-9a-f]+"|null),"span":"([0-9a-f]+)".*/\1 trace=\2 parent=\3 span=\4/'
   

   C trace=4bf92f3577b34da6a3ce929d0e0e4736 parent="23da5af9de835a5e" span=5daecad284d465a0
   B trace=4bf92f3577b34da6a3ce929d0e0e4736 parent="15b335cfa1b1e607" span=23da5af9de835a5e
   A trace=4bf92f3577b34da6a3ce929d0e0e4736 parent="00f067aa0ba902b7" span=15b335cfa1b1e607

   Read it bottom up. A's span is 15b335..., which is B's parent. B's span is 23da5a..., which is C's parent. One trace id, three distinct spans, each one pointing at the one above it.

4. 4

   Step 4.

   Stop the chain, start it again with FORWARD_UNCHANGED=1 node chain.mjs > chain.log 2>&1 &, and send the same request.

   curl -s -o /dev/null -w 'status %{http_code}\n' -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' http://127.0.0.1:19731/
   

   status 200

5. 5

   Step 5.

   Render the parent chain again and compare it with step 3.

   grep '"tp"' chain.log | sed -E 's/.*"svc":"([A-C])".*"traceId":"([0-9a-f]+)","parentSpan":("[0-9a-f]+"|null),"span":"([0-9a-f]+)".*/\1 trace=\2 parent=\3 span=\4/'
   

   C trace=4bf92f3577b34da6a3ce929d0e0e4736 parent="93409645b3cc6c7e" span=330f6874572587c0
   B trace=4bf92f3577b34da6a3ce929d0e0e4736 parent="93409645b3cc6c7e" span=0da8e214509a067e
   A trace=4bf92f3577b34da6a3ce929d0e0e4736 parent="00f067aa0ba902b7" span=93409645b3cc6c7e

   The trace id is still the same at all three hops. B and C now name the same parent, 93409645b3cc6c7e, which is A's span. Nothing in the log points at B's span at all, so a backend draws B and C as siblings under A and the call from B to C vanishes.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | One trace id, a different span at each hop, each parent matching the span above | Propagation is correct | Nothing. This is the shape in step 3. | | One trace id, two hops naming the same parent | A service forwards the header unchanged | Fix the client in the service whose span nobody points at. It is the one that did not mint a span. | | A different trace id after one hop | The receiver rejected or ignored the header | Read the header it received. restarted: true says it rejected it, and the fix is upstream. | | parentSpan: null on a service that was called | The header never arrived | Look at what sits between the two services. A proxy that strips headers produces this. | | Trace ids match but the backend still shows broken traces | Span ids collide or are reused | Check that each outgoing request gets a fresh parent id, not one per process or per connection. |

Common mistakes

What to check next

• Traceparent header format: if the header is malformed, the receiver restarts the trace before propagation is even in question.
• Traceparent and tracestate: the second header travels with this one and has its own forwarding rules.
• Trace id in logs: propagation is only useful once each service writes the trace id into its log lines.
• How to check correlation id in logs: the application level identifier to fall back on while trace context is being fixed.

FAQ

Does OpenTelemetry handle propagation automatically?

Its instrumentation does it for the HTTP clients it patches. A hand written fetch call, a queue publish, or a client the agent does not recognise still sends nothing. Auto instrumentation covers most calls rather than all of them, which is why this check exists.

What must change between hops and what must not?

The trace id must not change. The parent id must, on every outgoing request, to the id of the span making it. The flags may change if the service takes its own sampling decision.

Is matching trace ids enough to prove propagation?

No, and step 5 is the counterexample. Both runs carry one trace id across three services. Only the parent ids tell them apart, so the assertion has to cover the span structure.

What should a service do when no traceparent arrives?

Create a trace id and a span id of its own and become the root. Correct for an entry point, a defect anywhere else, which is why parentSpan: null on an internal service points at whatever sits in front of it.

Can I check this without a tracing backend?

Yes. Everything above reads log lines. A backend renders the tree more comfortably, but the parent links it draws from are already in the logs, and reading them directly removes the backend's own sampling from the picture.

Verified