How to check JSON encoding

JSON has one encoding. RFC 8259 requires UTF-8 for text exchanged between systems and defines no charset parameter for application/json, so the header cannot answer this question. Read the bytes instead: curl -s url | xxd -p. A response that starts efbbbf, or that fails a UTF-8 decode, is the defect.

Why check this

Run this on every API that returns names, addresses or free text, and again after a gateway, a serializer or a logging layer is added between the service and the client. JSON looks encoding-free because the parser rarely complains, and that is the problem: the text arrives damaged and the response still validates against its schema.

Two concrete failures. A file exported with a BOM stops parsing in the consumer that reads it from disk. A service that sets charset=windows-1251 and means it sends bytes no standard client will decode that way, and every client turns the text into replacement characters without raising anything.

Prerequisites

• Node 22 and curl 8 or later.
• RFC 8259 section 8.1 for the encoding rule, and section 11 for the media type registration.
• Four fixtures: the same object as raw UTF-8, as ASCII with escapes, with a BOM, and in windows-1251.

mkdir json && cd json
node -e "require('fs').writeFileSync('utf8.json', JSON.stringify({ city: 'Тест' }))"
node mkescaped.mjs
iconv -f UTF-8 -t WINDOWS-1251 utf8.json > cp1251.json
printf '\xef\xbb\xbf' > bom.json && cat utf8.json >> bom.json
cd ..

import { readFileSync, writeFileSync } from 'node:fs';
const raw = readFileSync('utf8.json', 'utf8');
const ascii = raw.replace(/[^\x20-\x7e]/g, (c) => '\\u' + c.charCodeAt(0).toString(16).padStart(4, '0'));
writeFileSync('escaped.json', ascii, 'ascii');

• A server that hands those four files out, one per route, on port 8979. The charset on the last route is there to be ignored.

import { createServer } from 'node:http';
import { readFileSync } from 'node:fs';
const routes = {
  '/utf8':    ['application/json', readFileSync('json/utf8.json')],
  '/escaped': ['application/json', readFileSync('json/escaped.json')],
  '/bom':     ['application/json', readFileSync('json/bom.json')],
  '/cp1251':  ['application/json; charset=windows-1251', readFileSync('json/cp1251.json')],
};
createServer((req, res) => {
  const r = routes[req.url];
  if (!r) { res.writeHead(404).end(); return; }
  res.writeHead(200, { 'content-type': r[0], 'content-length': r[1].length });
  res.end(r[1]);
}).listen(8979, '127.0.0.1', () => console.log('json fixture server on http://127.0.0.1:8979'));

Steps

1. 1

   Step 1.

   Read the declared type and the actual bytes of each response.

   for r in utf8 escaped bom cp1251; do printf '%-9s %-40s %s\n' "/$r" "$(curl -sI http://127.0.0.1:8979/$r | grep -i '^content-type' | tr -d '\r' | sed 's/^content-type: //')" "$(curl -s http://127.0.0.1:8979/$r | xxd -p | tr -d '\n')"; done
   

   /utf8     application/json                         7b2263697479223a22d0a2d0b5d181d182227d
   /escaped  application/json                         7b2263697479223a225c75303432325c75303433355c75303434315c7530343432227d
   /bom      application/json                         efbbbf7b2263697479223a22d0a2d0b5d181d182227d
   /cp1251   application/json; charset=windows-1251   7b2263697479223a22d2e5f1f2227d

   7b is {. The third response opens with a BOM instead. The fourth is 15 bytes where the first is 19, because four Cyrillic letters became four single bytes.

2. 2

   Step 2.

   Ask a standard client what it decoded, and print code points rather than rendered text.

   const cps = (s) => [...s].map((c) => 'U+' + c.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')).join(' ');
   for (const r of ['utf8', 'escaped', 'bom', 'cp1251']) {
     const res = await fetch(`http://127.0.0.1:8979/${r}`);
     const ct = res.headers.get('content-type');
     let line;
     try {
       const body = await res.json();
       line = `city "${body.city}"  ${cps(body.city)}`;
     } catch (e) {
       line = `res.json() threw: ${e.message}`;
     }
     console.log(`/${r.padEnd(8)} ${ct.padEnd(38)} ${line}`);
   }
   

   /utf8     application/json                       city "Тест"  U+0422 U+0435 U+0441 U+0442
   /escaped  application/json                       city "Тест"  U+0422 U+0435 U+0441 U+0442
   /bom      application/json                       city "Тест"  U+0422 U+0435 U+0441 U+0442
   /cp1251   application/json; charset=windows-1251 city "����"  U+FFFD U+FFFD U+FFFD U+FFFD

   The declared charset=windows-1251 changed nothing. The client decoded as UTF-8, replaced the four bytes it could not read, and returned an object with no error at all.

3. 3

   Step 3.

   Read the same BOM bytes from disk instead of from the wire.

   import { readFileSync } from 'node:fs';
   const f = 'json/bom.json';
   const buf = readFileSync(f);
   console.log(`${f}  ${buf.length} bytes  first bytes ${[...buf.subarray(0, 3)].map((b) => b.toString(16)).join(' ')}`);
   try {
     const o = JSON.parse(readFileSync(f, 'utf8'));
     console.log(`  JSON.parse(readFileSync(f, "utf8"))   ok, city ${o.city}`);
   } catch (e) {
     console.log(`  JSON.parse(readFileSync(f, "utf8"))   ${e.name}: ${e.message}`);
   }
   try {
     const o = JSON.parse(new TextDecoder('utf-8').decode(buf));
     console.log(`  JSON.parse(new TextDecoder().decode(b))   ok, city ${o.city}`);
   } catch (e) {
     console.log(`  JSON.parse(new TextDecoder().decode(b))   ${e.name}: ${e.message}`);
   }
   

   json/bom.json  22 bytes  first bytes ef bb bf
   JSON.parse(readFileSync(f, "utf8"))   SyntaxError: Unexpected token '', "{"city":"Тест"}" is not valid JSON
   JSON.parse(new TextDecoder().decode(b))   ok, city Тест

   Same bytes, same runtime, one failure and one success. Step 2 read the identical file over HTTP and parsed it, because res.json() decodes through TextDecoder, which drops a leading BOM.

4. 4

   Step 4.

   Confirm that an escape and a raw character are two spellings of one value.

   import { readFileSync } from 'node:fs';
   const raw = JSON.parse(readFileSync('json/utf8.json', 'utf8')).city;
   const esc = JSON.parse(readFileSync('json/escaped.json', 'utf8')).city;
   const cps = (s) => [...s].map((c) => 'U+' + c.codePointAt(0).toString(16).toUpperCase()).join(' ');
   console.log(`utf8.json     19 bytes on disk, parses to ${cps(raw)}`);
   console.log(`escaped.json  35 bytes on disk, parses to ${cps(esc)}`);
   console.log(`equal strings: ${raw === esc}`);
   

   utf8.json     19 bytes on disk, parses to U+422 U+435 U+441 U+442
   escaped.json  35 bytes on disk, parses to U+422 U+435 U+441 U+442
   equal strings: true

   Different byte counts, identical strings after parsing. No consumer can tell which spelling the producer used.

5. 5

   Step 5.

   Send both spellings through a layer that decodes as windows-1252 and re-encodes as UTF-8, which is what a mis-configured gateway does.

   import { readFileSync } from 'node:fs';
   const through = (buf) => Buffer.from(new TextDecoder('windows-1252').decode(buf), 'utf8');
   for (const f of ['json/utf8.json', 'json/escaped.json']) {
     const before = readFileSync(f);
     const after = through(before);
     const city = JSON.parse(after.toString('utf8')).city;
     console.log(`${f.padEnd(19)} ${String(before.length).padStart(2)} bytes in, ${String(after.length).padStart(2)} out, identical ${before.equals(after)}  city after "${city}"`);
   }
   

   json/utf8.json      19 bytes in, 28 out, identical false  city after "ТеÑÑ‚"
   json/escaped.json   35 bytes in, 35 out, identical true  city after "Тест"

   The ASCII-escaped document came out byte for byte identical and still means Тест. The raw one gained nine bytes and lost its text.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | The body starts 7b or 5b | The response opens with { or [ | Nothing. This is what a JSON body looks like. | | The body starts efbbbf | A BOM was added before the JSON | Remove it at the producer. RFC 8259 forbids it on transmitted JSON text. | | U+FFFD in a parsed value | Bytes were not valid UTF-8 and were replaced | The text is already lost. Fix the producer, then re-fetch. | | charset= on application/json | A parameter with no defined meaning | Ignore it as a signal, and read it as a hint that somebody expected it to work. |

Common mistakes

What to check next

• How to check if API returns valid JSON: the structural check this one sits on top of.
• How to check content-type of API response: what the header is good for once you know charset is not part of it.
• How to check the encoding of a file: the same question for a JSON file at rest, with the ruling-out method.
• How to check if a page is UTF-8: HTML, where the charset parameter does decide the outcome.
• How to check for garbled characters from the wrong encoding: naming the encoding pair behind a string like ТеÑÑ‚.

FAQ

How do I check the charset of a Content-Type header?

curl -sI url | grep -i content-type. For text/html the parameter decides how the browser decodes the page. For application/json it decides nothing, which step 2 shows with a client that ignored it.

Should a JSON API send application/json; charset=utf-8?

It is harmless and it is not a fix. Compliant recipients ignore the parameter. If text is arriving damaged, the bytes are wrong, and a header cannot repair bytes.

Are unicode escapes in JSON safe to use?

Yes, and they are more durable. They parse to the same string as the literal character, and an ASCII-only document passes unchanged through a layer that would corrupt raw UTF-8, as measured in step 5.

How do I remove a BOM from a JSON file?

tail -c +4 file.json > fixed.json, then confirm the first byte is 7b. Better, stop the producer from writing one. RFC 8259 says implementations must not add a byte order mark to networked-transmitted JSON text.

Verified