Traceparent and tracestate

tracestate carries one comma separated entry per vendor beside traceparent. Send both headers into the chain and read the log: each service rewrites its own key, moves it to the front of the list and leaves the other entries where they were. Above 32 entries the last one is dropped.

Why check this

traceparent says which trace a request belongs to. tracestate is where each tracing system keeps its position in that trace, and it decides whether two vendors can follow one request. Losing an entry costs nothing visible in a response and everything in the other vendor's trace view.

Run this when a second observability product is introduced, when a gateway or mesh joins a request path, and after a change to the middleware writing trace headers. The failure it prevents: a proxy rebuilds tracestate from the entries it recognises, the other vendor's entry disappears at the first hop, and that vendor shows every request as a new trace starting mid path.

Everything below runs on one machine against three services you start, so it tests header handling and nothing else.

Prerequisites

Node 22 and no packages. Output below came from v22.23.2.
curl. The output below came from 8.21.0, and any build sends these headers.
The W3C Trace Context recommendation of 23 November 2021. Section 3.3 defines the list grammar and section 3.5 the mutation rules quoted here.
Three chained services, A calling B calling C. Each rewrites its own tracestate entry under the key svca, svcb or svcc and logs the list it received and the list it sent. Save as chain.mjs.

import { createServer } from 'node:http';
import { randomBytes } from 'node:crypto';

const PORT = { A: 19731, B: 19732, C: 19733 };
const TP = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;

function parseTraceparent(h) {
  const m = typeof h === 'string' && TP.exec(h);
  if (!m) return null;
  if (/^0+$/.test(m[1]) || /^0+$/.test(m[2])) return null;
  return { traceId: m[1], parentId: m[2], flags: m[3] };
}

function mutateTracestate(raw, key, value) {
  const kept = (raw ?? '').split(',').map((s) => s.trim()).filter(Boolean)
    .filter((e) => e.split('=')[0] !== key);
  const list = [`${key}=${value}`, ...kept];
  return { header: list.slice(0, 32).join(','), dropped: list.slice(32) };
}

function service(name, port, next) {
  const key = 'svc' + name.toLowerCase();
  createServer(async (req, res) => {
    const tpIn = req.headers.traceparent ?? null;
    const tsIn = req.headers.tracestate ?? null;
    const ctx = parseTraceparent(tpIn) ?? { traceId: randomBytes(16).toString('hex'), parentId: null, flags: '01', restarted: true };
    const span = randomBytes(8).toString('hex');
    const ts = mutateTracestate(ctx.restarted ? null : tsIn, key, span);
    let tpOut = null;
    const body = { svc: name };
    if (next) {
      tpOut = process.env.FORWARD_UNCHANGED === '1' && name === 'B' ? tpIn : `00-${ctx.traceId}-${span}-${ctx.flags}`;
      const r = await fetch(`http://127.0.0.1:${next}/`, { headers: { traceparent: tpOut, tracestate: ts.header } });
      body.downstream = await r.json();
    }
    console.log(JSON.stringify({ kind: 'tp', svc: name, in: tpIn, traceId: ctx.traceId, parentSpan: ctx.parentId, span, restarted: Boolean(ctx.restarted), out: tpOut }));
    console.log(JSON.stringify({ kind: 'ts', svc: name, in: tsIn, out: ts.header, dropped: ts.dropped }));
    res.setHeader('content-type', 'application/json');
    res.end(JSON.stringify(body));
  }).listen(port, '127.0.0.1');
}

service('C', PORT.C, null);
service('B', PORT.B, PORT.C);
service('A', PORT.A, PORT.B);
console.log(`A ${PORT.A} -> B ${PORT.B} -> C ${PORT.C}`);

Each scenario below is one request. Start the chain fresh before steps 1, 3, 5 and 7 with rm -f chain.log && node chain.mjs > chain.log 2>&1 &, so a grep returns one request.
Stop it afterwards: find the process id with netstat -ano | grep 19731 and stop that id alone.

Steps

Send a request carrying both headers, with two entries belonging to other vendors.

curl -s -o /dev/null -w 'status %{http_code}\n' -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' -H 'tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7' http://127.0.0.1:19731/

status 200

Read the list each service received and the one it sent on.

grep '"ts"' chain.log

{"kind":"ts","svc":"C","in":"svcb=e3be9471a18a6b0b,svca=b3f38825d0c86401,congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","out":"svcc=b4c32871ebc95c86,svcb=e3be9471a18a6b0b,svca=b3f38825d0c86401,congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","dropped":[]}
{"kind":"ts","svc":"B","in":"svca=b3f38825d0c86401,congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","out":"svcb=e3be9471a18a6b0b,svca=b3f38825d0c86401,congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","dropped":[]}
{"kind":"ts","svc":"A","in":"congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","out":"svca=b3f38825d0c86401,congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","dropped":[]}

Each service added its own key at the front, congo and rojo kept their order to C, and the list grew by one entry per hop. That growth is why the ceiling in step 6 is reachable.

Restart the chain and send the same request with a stale svca entry in the middle of the list.

curl -s -o /dev/null -w 'status %{http_code}\n' -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' -H 'tracestate: congo=t61rcWkgMzE,svca=0000000000000001,rojo=00f067aa0ba902b7' http://127.0.0.1:19731/

status 200

4
Step 4.
Read what service A did with the entry carrying its own key.
```
grep '"ts","svc":"A"' chain.log
```
```
{"kind":"ts","svc":"A","in":"congo=t61rcWkgMzE,svca=0000000000000001,rojo=00f067aa0ba902b7","out":"svca=5d6ec543f93bd703,congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","dropped":[]}
```
The old svca=0000000000000001 is gone and a new svca sits at the front, congo and rojo in their original order behind it. One entry per key, most recent writer first.

Restart the chain and send a list already holding the maximum of 32 entries.

TS=$(node -e "console.log(Array.from({length:32},(_,i)=>'v'+(i+1)+'=x'+(i+1)).join(','))")
curl -s -o /dev/null -w 'status %{http_code}\n' -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' -H "tracestate: $TS" http://127.0.0.1:19731/

status 200

Read which entry left the list when service A added its own.

grep '"ts","svc":"A"' chain.log

{"kind":"ts","svc":"A","in":"v1=x1,v2=x2,v3=x3,v4=x4,v5=x5,v6=x6,v7=x7,v8=x8,v9=x9,v10=x10,v11=x11,v12=x12,v13=x13,v14=x14,v15=x15,v16=x16,v17=x17,v18=x18,v19=x19,v20=x20,v21=x21,v22=x22,v23=x23,v24=x24,v25=x25,v26=x26,v27=x27,v28=x28,v29=x29,v30=x30,v31=x31,v32=x32","out":"svca=90ed2707be56d00c,v1=x1,v2=x2,v3=x3,v4=x4,v5=x5,v6=x6,v7=x7,v8=x8,v9=x9,v10=x10,v11=x11,v12=x12,v13=x13,v14=x14,v15=x15,v16=x16,v17=x17,v18=x18,v19=x19,v20=x20,v21=x21,v22=x22,v23=x23,v24=x24,v25=x25,v26=x26,v27=x27,v28=x28,v29=x29,v30=x30,v31=x31","dropped":["v32=x32"]}

The received list held v1 to v32. The sent list holds 32 again: svca at the front, v1 to v31 behind. v32 left, and its owner receives no notice.

Restart the chain and send a valid tracestate beside an invalid traceparent.

curl -s -o /dev/null -w 'status %{http_code}\n' -H 'traceparent: 00-00000000000000000000000000000000-00f067aa0ba902b7-01' -H 'tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7' http://127.0.0.1:19731/

status 200

8
Step 8.
Read what happened to the entries that were well formed.
```
grep '"ts","svc":"A"' chain.log
```
```
{"kind":"ts","svc":"A","in":"congo=t61rcWkgMzE,rojo=00f067aa0ba902b7","out":"svca=80401f1b911352e1","dropped":[]}
```
Both entries went because the trace id beside them was all zeroes. A receiver that cannot parse traceparent deletes tracestate with it, so one broken producer erases every vendor's state.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Your key at the front, other keys behind in their old order | Correct mutation | Nothing. This is steps 2 and 4. | | Your key updated in place, further down the list | The order rule is not followed | Move the updated entry to the front. Order is how a reader tells recent writers from old ones. | | Two entries with the same key | The writer appends instead of replacing | Replace. One entry per key, because the entry records the last position in that system. | | dropped is not empty | The list hit 32 entries | Find out whose entry left. Truncation removes whole entries, never part of one. | | tracestate empty while the request carried one | traceparent failed to parse at this hop | Fix the producer of traceparent. The state was collateral damage. | | Another vendor's key missing with no truncation | Something deleted a key it does not own | Look at proxies and middleware. Deleting a foreign key breaks correlation in that vendor's backend. |

Common mistakes

Sign: tracestate arrives empty at a service although the caller sent entries, and nothing reports an error.Cause: The rule is that a receiver which fails to parse traceparent deletes tracestate as well. Step 8 shows two valid vendor entries being discarded because the trace id beside them was all zeroes. Debugging starts at whatever wrote the broken traceparent, not at the service that lost the state.

Sign: A proxy rebuilds tracestate from the keys it knows and the trace looks fine in your own backend.Cause: Deleting a key a service did not generate is what the recommendation tells vendors not to do, because the loss is invisible to the deleter and total for the owner. Your vendor's entry survives every hop, so your traces look healthy while the other product sees a new trace starting mid path.

Sign: Entries disappear from the end of a long list under load and reappear on quieter requests.Cause: The list has room for 32 members, and every hop that writes an entry uses one. A path of five services with a gateway and a mesh sidecar reaches the ceiling on exactly the requests that took the longest route, which is the set worth tracing.

Thresholds

32 list members maximum, and at least 512 characters of the combined header should be propagated Source: https://www.w3.org/TR/trace-context/#tracestate-limits

The two numbers bind different things. The grammar permits 32 members and no more, which step 6 exercises. The 512 characters is a floor rather than a cap: carry at least that much of the combined header, and document any lower limit. Truncation removes whole entries, those over 128 characters first.

What to check next

Traceparent header format: the header whose failure to parse takes tracestate with it in step 8.
How to check trace propagation between services: the identity half of trace context, and the one that decides the shape of the trace.
Trace id in logs: what to assert once both headers survive the path.
How to check correlation id in logs: the application level identifier that is unaffected by vendor state.

FAQ

What does a tracestate header look like?

tracestate: congo=t61rcWkgMzE,rojo=00f067aa0ba902b7. Each member is a lowercase key, an equals sign and an opaque value, separated by commas. A key may carry a tenant prefix, as in 1234@congo, so a vendor finds its own entries by searching for @congo=.

How is tracestate different from traceparent?

traceparent has a fixed grammar and one meaning, shared by everybody. tracestate is a list in which each vendor owns one key whose value is opaque to the others.

What happens when the list is full?

The vendor adding an entry removes one. Step 6 shows v32 leaving as svca arrives, and a removal always takes a whole entry.

Can a service delete another vendor's entry?

It can, and it should not. Deletion exists so proxies can strip entries for privacy and so oversized lists can be truncated. Outside those cases it breaks correlation in a system you cannot see.

Verified

Verified 2026-09-12by Maks Vernynode 22.23.2curl 8.21.0

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

All logging and observability checkscluster

intermediate8 minpublished 2026-09-12updated 2026-09-12Maks Verny