Prometheus text exposition format

Read the body as a grammar, not as a log. Each name gets one # HELP and one # TYPE line, each sample line needs a unique name and label set, and histogram buckets must be cumulative, sorted, and closed by +Inf. Running node histogram-lint.mjs against a hand-rolled exporter broke three of those rules.

Why check this

Run this when a team writes its own exporter, or formats metrics in a template, instead of calling an instrumentation library. That code produces text that looks right, scrapes without complaint, and yields wrong quantiles for months.

The failure it prevents is a latency dashboard that answers with confidence and is wrong. A histogram whose buckets hold per-bucket counts instead of cumulative ones still charts. histogram_quantile reads it as a valid distribution and returns a p95 nobody can reproduce from the logs.

Prerequisites

• Node 22 and curl. Both commands run against a loopback port.
• npm i parse-prometheus-text-format. Version 1.1.1 is used below.
• The exposition formats page is the source for every rule stated here. Prometheus itself is not installed on the machine that produced this page, so the outputs show what one widely used parser does, and each rule is attributed to the documentation rather than to a scrape.
• A local target serving four bodies. Save this as format-lab.mjs and start it with node format-lab.mjs.

// format-lab.mjs  node 22, ESM. Four exposition bodies on one port.
import { createServer } from 'node:http';

const queue = [1200, 1430, 12, 40];
let scrape = 0;

const bodies = {
  '/good': `# HELP request_duration_seconds Time spent in the handler.
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.1"} 12
request_duration_seconds_bucket{le="0.5"} 19
request_duration_seconds_bucket{le="1"} 23
request_duration_seconds_bucket{le="+Inf"} 23
request_duration_seconds_sum 9.84
request_duration_seconds_count 23
`,
  '/dup': `# HELP orders_total Orders accepted.
# TYPE orders_total counter
# TYPE orders_total gauge
orders_total{region="eu"} 12
orders_total{region="eu"} 31
`,
  '/hist': `# HELP request_duration_seconds Time spent in the handler.
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.5"} 7
request_duration_seconds_bucket{le="0.1"} 12
request_duration_seconds_bucket{le="1"} 4
request_duration_seconds_sum 9.84
request_duration_seconds_count 23
`,
};

createServer((req, res) => {
  const path = req.url.split('?')[0];
  const body = path === '/counter'
    ? `# HELP queue_jobs_total Jobs taken off the queue.\n# TYPE queue_jobs_total counter\nqueue_jobs_total ${queue[scrape++ % queue.length]}\n`
    : bodies[path];
  res.writeHead(body ? 200 : 404, { 'Content-Type': 'text/plain; version=0.0.4; charset=utf-8' });
  res.end(body ?? 'no such fixture\n');
}).listen(39118, '127.0.0.1', () => console.log('format lab on 127.0.0.1:39118'));

• The bucket checker, saved as histogram-lint.mjs.

// histogram-lint.mjs <url> : the three bucket rules plus the _count identity.
const num = (s) => (s === '+Inf' ? Infinity : Number(s)); // Number('+Inf') is NaN
const body = await (await fetch(process.argv[2])).text();
const buckets = [], counts = new Map();
for (const line of body.split('\n')) {
  let m = line.match(/^(\w+)_bucket\{le="([^"]+)"\} (\S+)/);
  if (m) buckets.push({ name: m[1], le: num(m[2]), value: Number(m[3]) });
  m = line.match(/^(\w+)_count (\S+)/);
  if (m) counts.set(m[1], Number(m[2]));
}
for (const name of new Set(buckets.map((b) => b.name))) {
  const b = buckets.filter((x) => x.name === name);
  const inf = b.find((x) => x.le === Infinity);
  console.log(`${name}: ${b.length} buckets, _count ${counts.get(name)}`);
  console.log(`  +Inf bucket present: ${Boolean(inf)}`);
  console.log(`  le ascending:        ${b.every((x, i) => i === 0 || x.le > b[i - 1].le)}`);
  console.log(`  values cumulative:   ${b.every((x, i) => i === 0 || x.value >= b[i - 1].value)}`);
  console.log(`  +Inf equals _count:  ${inf ? inf.value === counts.get(name) : 'no +Inf bucket'}`);
}

Steps

1. 1

   Step 1.

   Read a correct histogram and name every line in it.

   curl -s http://127.0.0.1:39118/good
   

   # HELP request_duration_seconds Time spent in the handler.
   # TYPE request_duration_seconds histogram
   request_duration_seconds_bucket{le="0.1"} 12
   request_duration_seconds_bucket{le="0.5"} 19
   request_duration_seconds_bucket{le="1"} 23
   request_duration_seconds_bucket{le="+Inf"} 23
   request_duration_seconds_sum 9.84
   request_duration_seconds_count 23

   One HELP, one TYPE, then the samples for that name grouped together. The bucket values rise because each bucket counts every observation at or below its le.

2. 2

   Step 2.

   Run the bucket rules against that body to see what passing looks like.

   node histogram-lint.mjs http://127.0.0.1:39118/good
   

   request_duration_seconds: 4 buckets, _count 23
   +Inf bucket present: true
   le ascending:        true
   values cumulative:   true
   +Inf equals _count:  true

   The documentation requires the +Inf bucket and states its value must be identical to x_count. Both hold at 23.

3. 3

   Step 3.

   Run the same rules against the hand-rolled version.

   node histogram-lint.mjs http://127.0.0.1:39118/hist
   

   request_duration_seconds: 3 buckets, _count 23
   +Inf bucket present: false
   le ascending:        false
   values cumulative:   false
   +Inf equals _count:  no +Inf bucket

   Three faults in five lines: no +Inf, buckets emitted as 0.5, 0.1, 1, and values 7, 12, 4 that are per-bucket counts rather than cumulative ones. Nothing about this body fails a scrape.

4. 4

   Step 4.

   Read a body that repeats a series and declares its type twice.

   curl -s http://127.0.0.1:39118/dup
   

   # HELP orders_total Orders accepted.
   # TYPE orders_total counter
   # TYPE orders_total gauge
   orders_total{region="eu"} 12
   orders_total{region="eu"} 31

   Only one TYPE line may exist for a given metric name, and each sample line must have a unique combination of name and labels. This body breaks both rules.

5. 5

   Step 5.

   Parse it and look at what survives.

   node -e "const parse = require('parse-prometheus-text-format'); fetch('http://127.0.0.1:39118/dup').then((r) => r.text()).then((b) => console.log(JSON.stringify(parse(b))));"
   

   [{"name":"orders_total","help":"Orders accepted.","type":"COUNTER","metrics":[{"value":"31","labels":{"region":"eu"}}]}]

   One series, value 31. The 12 is gone, the second TYPE line was ignored, and no error was raised. The documentation calls ingestion of a repeated name and label set undefined, which is what undefined looks like from the outside.

6. 6

   Step 6.

   Scrape a counter four times and read the sequence.

   for i in 1 2 3 4; do curl -s http://127.0.0.1:39118/counter | grep -v '^#'; done
   

   queue_jobs_total 1200
   queue_jobs_total 1430
   queue_jobs_total 12
   queue_jobs_total 40

   The third scrape is lower than the second. Serving that is legal. Reading it as a counter is not.

7. 7

   Step 7.

   Apply the documented reset correction to those four samples.

   node -e "const v=[1200,1430,12,40];let c=0;for(let i=1;i<v.length;i++)if(v[i]<v[i-1])c+=v[i-1];console.log('naive last-first:',v.at(-1)-v[0]);console.log('reset-corrected:',v.at(-1)+c-v[0]);"
   

   naive last-first: -1160
   reset-corrected: 270

   rate and increase adjust for breaks in monotonicity, so the drop is read as a restart and the pre-drop value is added back. The chart shows 270 jobs of work over a window in which the counter ended below where it started.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Two # TYPE lines for one name | The exporter writes headers per sample, not per name | Emit one HELP and one TYPE, then the group of samples. | | The same name and labels on two lines | Ingestion is undefined by the documentation | Deduplicate at the source. The parser used here kept the last value silently. | | No +Inf bucket | The histogram has no closing bucket | Add it, with the same value as x_count. | | Bucket values that fall as le rises | Buckets hold per-bucket counts | Make each bucket the running total of everything at or below its le. | | A counter that decreases between scrapes | A gauge is exposed as a counter, or the process restarted | Change the type, or confirm a restart at that timestamp before trusting the rate. |

Common mistakes

What to check next

• How to check metrics endpoint: the read before this one, including the Content-Type and the series count.
• Prometheus metrics endpoint authentication: what this body discloses to whoever can fetch it.
• Json logging format: the same discipline applied to the log stream.
• How to test a health check endpoint: the neighbouring endpoint with its own contract.

FAQ

What types of metrics are there in Prometheus?

Four: counter, gauge, histogram and summary. The metric types page defines them. The exposition format also allows untyped for a name whose type the exporter does not declare.

Does the exposition need a trailing newline?

Yes. The documentation states that the last line must end with a line feed character. A body assembled with join and no final separator omits it.

Is a counter allowed to go down?

The text format will serve any float. A decrease is read as a counter reset, and step 7 shows the effect: an increase of 270 over a window that ended 1160 lower than it started.

Can I put a user id in a label?

Not in a metric that a scrape collects. Every distinct value creates a series. Count series per name before shipping the label.

How do I check the format without installing Prometheus?

Fetch the body and read it. The exposition format is text, so a parser and a few rules cover it, which is what the two scripts above do. Nothing on this page needs a running Prometheus server.

Verified