Trace id in logs

Replay one request with a known traceparent, then search every service log for the 32 character trace id with grep -h '4bf92f3577b34da6a3ce929d0e0e4736' gateway.log stock.log. Each service logs that same trace id and a span id of its own. A line carrying a different trace id means the header was rejected.

Why check this

Run this when tracing is introduced, when a service is rewritten, and after any change to the ingress. The failure it prevents is the one that wastes an incident: the tracing backend shows a slow trace, the engineer copies its id into the log search, and nothing comes back, because the logs carry a different id or no id at all.

The check is narrow on purpose. It compares the id a service was given on the wire with the id it wrote to its log. It says nothing about whether spans reached a collector, and nothing about sampling decisions made upstream. Those are separate, and step 4 shows why the difference matters.

Prerequisites

• Node 22. Confirm with node --version.
• curl 8, any build. See the curl manual.
• gateway.mjs below, started with node gateway.mjs > gateway.log. It parses traceparent by hand so the whole mechanism is visible. A tracing SDK does the same work and puts the result on the active span context; the check does not change.

import { createServer } from 'node:http';
import { randomBytes } from 'node:crypto';

const hex = (n) => randomBytes(n).toString('hex');
const log = (o) => console.log(JSON.stringify({ ts: new Date().toISOString(), svc: 'gateway', ...o }));
const parse = (h) => {
  const m = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/.exec(h ?? '');
  if (!m || m[1] === '0'.repeat(32) || m[2] === '0'.repeat(16)) return null;
  return { trace: m[1], parent: m[2], flags: m[3] };
};

createServer(async (req, res) => {
  const ctx = parse(req.headers.traceparent);
  const trace_id = ctx?.trace ?? hex(16);
  const span_id = hex(8);
  const flags = ctx?.flags ?? '01';
  log({ level: 'info', msg: 'handle', path: req.url, trace_id, span_id,
        parent_span_id: ctx?.parent ?? null, sampled: flags === '01',
        source: ctx ? 'traceparent' : 'new' });
  const traceparent = `00-${trace_id}-${span_id}-${flags}`;
  const r = await fetch('http://127.0.0.1:9432/stock', { headers: { traceparent } });
  res.setHeader('traceparent', traceparent);
  res.end(JSON.stringify({ ok: true, stock: r.status }));
}).listen(9431, '127.0.0.1', () => log({ level: 'info', msg: 'listening', port: 9431 }));

• stock.mjs below, started with node stock.mjs > stock.log. It logs the trace id it received and mints a span id of its own.

import { createServer } from 'node:http';
import { randomBytes } from 'node:crypto';

const log = (o) => console.log(JSON.stringify({ ts: new Date().toISOString(), svc: 'stock', ...o }));

createServer((req, res) => {
  const m = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/.exec(req.headers.traceparent ?? '');
  log({ level: 'info', msg: 'lookup', trace_id: m ? m[1] : null,
        span_id: randomBytes(8).toString('hex'), parent_span_id: m ? m[2] : null });
  res.end(JSON.stringify({ ok: true }));
}).listen(9432, '127.0.0.1', () => log({ level: 'info', msg: 'listening', port: 9432 }));

Steps

1. 1

   Step 1.

   Call the front service with no traceparent and read the one it returns.

   curl -s -D - -o /dev/null http://127.0.0.1:9431/cart | grep -i traceparent
   

   traceparent: 00-9248af2a389a65c084071a89d08f8974-8bb4725f76303936-01

   The four fields are version, trace id, span id and flags. The trace id is the middle 32 characters. Where those characters come from is the subject of Traceparent header format.

2. 2

   Step 2.

   Search both logs for that trace id.

   grep -h 9248af2a389a65c084071a89d08f8974 gateway.log stock.log
   

   {"ts":"2026-09-12T07:28:12.935Z","svc":"gateway","level":"info","msg":"handle","path":"/cart","trace_id":"9248af2a389a65c084071a89d08f8974","span_id":"8bb4725f76303936","parent_span_id":null,"sampled":true,"source":"new"}
   {"ts":"2026-09-12T07:28:12.945Z","svc":"stock","level":"info","msg":"lookup","trace_id":"9248af2a389a65c084071a89d08f8974","span_id":"ce758c27e490591e","parent_span_id":"8bb4725f76303936"}

   One trace id, two span ids. The gateway span 8bb4725f76303936 is the parent_span_id of the stock line, so the two lines are ordered as well as joined. source: new says no caller supplied a trace, so the gateway started one.

3. 3

   Step 3.

   Replay a traceparent you supply and confirm the services adopt the trace id rather than inventing one.

   curl -s -o /dev/null -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' http://127.0.0.1:9431/cart; grep -h 4bf92f3577b34da6a3ce929d0e0e4736 gateway.log stock.log
   

   {"ts":"2026-09-12T07:28:27.968Z","svc":"gateway","level":"info","msg":"handle","path":"/cart","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"217fec733b5a2380","parent_span_id":"00f067aa0ba902b7","sampled":true,"source":"traceparent"}
   {"ts":"2026-09-12T07:28:27.971Z","svc":"stock","level":"info","msg":"lookup","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"8a8e49d1fccc763f","parent_span_id":"217fec733b5a2380"}

   source: traceparent is the passing result. The span id you sent has become the gateway's parent_span_id, and the gateway's own span id has become the parent of the stock span.

4. 4

   Step 4.

   Replay the same shape with the sampled flag cleared, -00 instead of -01.

   curl -s -o /dev/null -H 'traceparent: 00-9c1e2b4a55d34e7b8f0a1d2c3b4a5968-00f067aa0ba902b7-00' http://127.0.0.1:9431/cart; grep -h 9c1e2b4a55d34e7b8f0a1d2c3b4a5968 gateway.log
   

   {"ts":"2026-09-12T07:28:28.025Z","svc":"gateway","level":"info","msg":"handle","path":"/cart","trace_id":"9c1e2b4a55d34e7b8f0a1d2c3b4a5968","span_id":"dfbd13494d8c1251","parent_span_id":"00f067aa0ba902b7","sampled":false,"source":"traceparent"}

   The log line is complete and the trace was never recorded. Pasting this id into a tracing UI returns nothing, and the logs are still correct. Read sampled before concluding that tracing is broken.

5. 5

   Step 5.

   Replay a trace id in uppercase, which the specification treats as invalid.

   curl -s -o /dev/null -H 'traceparent: 00-4BF92F3577B34DA6A3CE929D0E0E4737-00f067aa0ba902b7-01' http://127.0.0.1:9431/cart; tail -1 gateway.log; grep -c -i 4BF92F3577B34DA6A3CE929D0E0E4737 gateway.log
   

   {"ts":"2026-09-12T07:28:34.326Z","svc":"gateway","level":"info","msg":"handle","path":"/cart","trace_id":"8e41ed82cc9a1d5956f93c8be7938172","span_id":"a0402017e1d98ef5","parent_span_id":null,"sampled":true,"source":"new"}
   0

   The request succeeded, the log line looks healthy, and the id the caller holds appears nowhere in the file. The final 0 is the count of case insensitive matches for the id that was sent.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | The same trace_id in every service log | Context propagates and logging reads it | Nothing. Record the id in the test report. | | source: new on a service behind the edge | That service ignored or never received the header | Compare with the header the caller sent, then look at the client, not the receiver. | | sampled: false on a line you cannot find in the tracing UI | The trace was never recorded upstream | Correct behaviour. Raise the sampling rate on the environment under test. | | trace_id: null in a log line | No traceparent arrived and the service does not start a trace | Decide which service owns trace creation, usually the ingress. | | Two services sharing a span_id | One of them logs its parent as its own span | Fix the instrumentation. Spans are per operation, never shared. |

Thresholds

Common mistakes

What to check next

• Traceparent header format: the four fields, their lengths, and what makes one invalid.
• How to check trace propagation between services: the hop itself, when a log line names only one service.
• Traceparent and tracestate: the vendor state that travels beside the trace id and is dropped more often.
• How to check correlation id in logs: the application level id that usually predates tracing and outlives it.
• Json logging format: a trace id is only searchable at scale when it is a field with a stable name.

FAQ

What is the difference between a trace id and a span id?

A trace id names the whole request across every service. A span id names one operation inside it. Step 2 shows one trace id on two lines and a different span id on each, with the caller's span recorded as parent_span_id.

What is the difference between a trace id, a span id and a correlation id?

Trace id and span id come from the tracing system and have fixed formats. A correlation id is invented by the application, has whatever shape the team chose, and often predates the tracing rollout. During a migration a log line carries both, and the two are searched separately.

How does the trace id get into a log line with OpenTelemetry?

The SDK keeps the active span context, and the logging bridge copies trace_id and span_id onto each record. The services here do that work by hand so the mechanism is visible. Whichever produces the field, this check reads the result rather than the configuration.

Why does the field have to be named trace_id?

It does not have to, but the name has to be identical in every service, or a single query cannot join them. Pick the name your log store and tracing UI already link on, then test it with one replayed request as in step 3.

Verified