h2check

How to check if a page is UTF-8

A page declares its encoding in three places: a BOM, the Content-Type header, and <meta charset>. Read all three with curl -sI and a grep over the body, then open the page and run document.characterSet in the Console. That last value is the encoding the browser used.

Why check this

Run this after any change to a template, a CMS export, a reverse proxy or a CDN rule, and on every page that carries names, addresses or free text from users. A proxy that appends charset=windows-1251 to Content-Type turns every accented character on a UTF-8 page into mojibake, and the template still says <meta charset="utf-8">, so a code review finds nothing.

The three declarations do not carry equal weight, and the page that renders correctly on your machine can render as garbage on a colleague's. The steps below measure the order.

Prerequisites

import { createServer } from 'node:http';
const doc = (head) => Buffer.from(
  `<!doctype html><html><head>${head}<title>encoding fixture</title></head>` +
  `<body><p id="s">Zurich cafe, Привет</p></body></html>`, 'utf8');
const bom = Buffer.from([0xef, 0xbb, 0xbf]);
const pad = `<!--${'x'.repeat(Number(process.env.PAD ?? 1100))}-->`;
const routes = {
  '/header-1251': ['text/html; charset=windows-1251', doc('<meta charset="utf-8">')],
  '/bom-1251':    ['text/html; charset=windows-1251', Buffer.concat([bom, doc('')])],
  '/meta-only':   ['text/html',                       doc('<meta charset="utf-8">')],
  '/meta-late':   ['text/html',                       doc(`${pad}<meta charset="utf-8">`)],
  '/nothing':     ['text/html',                       doc('')],
};
createServer((req, res) => {
  const r = routes[req.url];
  if (!r) { res.writeHead(404).end(); return; }
  res.writeHead(200, { 'content-type': r[0], 'content-length': r[1].length });
  res.end(r[1]);
}).listen(8971, '127.0.0.1', () => console.log('fixture server on http://127.0.0.1:8971'));

Steps

  1. Step 1.

    Read what the header declares on the page under test.

    curl -s -D - -o page.html --compressed https://developer.mozilla.org/en-US/docs/Web/HTML | grep -i '^content-type'

    content-type: text/html

    No charset parameter. The header has left the decision to the document.

  2. Step 2.

    Read what the document declares, from the body saved in step 1.

    head -c 1024 page.html | grep -o -i -E '<meta[^>]*charset[^>]*>'

    <meta charset="UTF-8" />

    One declaration, in the document, inside the first kilobyte.

  3. Step 3.

    Check a page that declares nothing at all, so you know what that looks like.

    curl -s https://example.com/ -o example.html && grep -o -i -E '<meta[^>]*>' example.html

    <meta name="viewport" content="width=device-width, initial-scale=1">

    The 559-byte document carries one meta element and it is not a charset. The header on the same response is content-type: text/html, with no parameter either. Nothing in this response says how to decode it.

  4. Step 4.

    Start the fixture server, then read all three declaration sites for each route in one pass.

    for r in header-1251 bom-1251 meta-only meta-late nothing; do printf '%-12s ' "$r"; h=$(curl -sI http://127.0.0.1:8971/$r | grep -i '^content-type' | tr -d '\r' | sed 's/^content-type: //'); b=$(curl -s http://127.0.0.1:8971/$r | head -c 3 | xxd -p); m=$(curl -s http://127.0.0.1:8971/$r | grep -b -o -i -E '<meta[^>]*charset[^>]*>' | head -1); printf 'header "%s"  first3 %s  meta %s\n' "$h" "$b" "${m:-none}"; done

    header-1251  header "text/html; charset=windows-1251"  first3 3c2164  meta 27:<meta charset="utf-8">
bom-1251     header "text/html; charset=windows-1251"  first3 efbbbf  meta none
meta-only    header "text/html"  first3 3c2164  meta 27:<meta charset="utf-8">
meta-late    header "text/html"  first3 3c2164  meta 1134:<meta charset="utf-8">
nothing      header "text/html"  first3 3c2164  meta none

    first3 efbbbf is a UTF-8 BOM. The number before each meta is its byte offset in the response.

  5. Step 5.

    Ask the browser which of those declarations it obeyed. Open each route in Chrome and read document.characterSet in the Console, or drive it from a Puppeteer session, which is how the output below was produced.

    import { open } from './session.mjs';
const s = await open();
try {
  console.log('chrome', await s.browser.version());
  for (const r of ['header-1251', 'bom-1251', 'meta-only', 'meta-late', 'nothing']) {
    await s.goto(`http://127.0.0.1:8971/${r}`);
    const out = await s.page.evaluate(() => ({
      charset: document.characterSet,
      text: document.getElementById('s').textContent,
    }));
    console.log(`${r.padEnd(12)} document.characterSet ${out.charset.padEnd(14)} text "${out.text}"`);
  }
} finally {
  await s.close();
}

    chrome Chrome/152.0.7977.76
header-1251  document.characterSet windows-1251   text "Zurich cafe, Привет"
bom-1251     document.characterSet UTF-8          text "Zurich cafe, Привет"
meta-only    document.characterSet UTF-8          text "Zurich cafe, Привет"
meta-late    document.characterSet UTF-8          text "Zurich cafe, Привет"
nothing      document.characterSet windows-1251   text "Zurich cafe, Привет"

    Every route served the same UTF-8 bytes. Row one lost to the header, row two won with a BOM under the same header, row five had nothing to go on.

  6. Step 6.

    Load the route that declares nothing under three browser languages, to see what the fallback costs.

    import { open } from './session.mjs';
for (const lang of ['uk-UA', 'en-US', 'ja-JP']) {
  const s = await open({ args: [`--lang=${lang}`] });
  try {
    await s.goto('http://127.0.0.1:8971/nothing');
    const out = await s.page.evaluate(() => ({
      cs: document.characterSet, nav: navigator.language,
    }));
    console.log(`--lang=${lang.padEnd(6)} navigator.language ${out.nav.padEnd(6)} document.characterSet ${out.cs}`);
  } finally { await s.close(); }
}

    --lang=uk-UA  navigator.language uk-UA  document.characterSet windows-1251
--lang=en-US  navigator.language en-US  document.characterSet windows-1252
--lang=ja-JP  navigator.language ja     document.characterSet Shift_JIS

    One byte stream, three decodings, chosen by the reader's browser language.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | document.characterSet is UTF-8 | The browser decoded the page as UTF-8 | Nothing. This is the value that matters, whichever declaration produced it. | | Header says charset=windows-1251, meta says utf-8 | The header wins and the text is mojibake | Fix the server or proxy. Changing the meta has no effect here. | | No charset in the header and no meta | The browser guesses from its own language | Add charset=utf-8 to the header, or a <meta charset> in the head. | | first3 efbbbf on an HTML response | The file carries a UTF-8 BOM | The BOM outranks the header. Remove it unless you meant it. |

Common mistakes

Sign: The template declares utf-8, the page still renders as mojibake, and the markup looks correct in every review.Cause: A charset in the Content-Type header outranks the meta element. The fixture above proves it: /header-1251 served UTF-8 bytes with a utf-8 meta and Chrome reported windows-1251. Read the header before reading the markup.
Sign: A reviewer rejects a meta charset because it is not in the first 1024 bytes.Cause: 1024 bytes is the prescan window, not a cutoff. Chrome 152 honoured the meta at byte offset 1134 on /meta-late, and at 100034 when the fixture ran with PAD=100000, by restarting the parse. The restart is real work thrown away, so keep the declaration early, but the page is not broken by the offset alone.
Sign: The page renders correctly for the team and turns to garbage for one customer.Cause: With no declaration anywhere, the fallback follows the browser language. The same response decoded as windows-1251, windows-1252 and Shift_JIS under three Chrome languages in step 6.
Sign: Content-Encoding: gzip is read as an answer about character encoding.Cause: Content-Encoding describes compression, charset describes text decoding. They share a word and nothing else. A gzip response can carry any charset, and neither implies the other.

Thresholds

1024 bytes Source: HTML Standard, prescan a byte stream to determine its encoding

That is how far the browser reads before it starts parsing. A <meta charset> inside that window costs nothing. Past it, Chrome 152 still applied the declaration in this capture, at the cost of a restarted parse.

What to check next

FAQ

How do I check meta charset without DevTools?

Fetch the first kilobyte and grep it: curl -s https://example.com/ | head -c 1024 | grep -o -i 'charset[^>]*'. That tells you what the document declares. It does not tell you what the browser used, because a header or a BOM can outrank it.

Which declaration wins?

In the capture above: BOM, then the Content-Type header, then <meta charset>, then the browser language. /bom-1251 rendered as UTF-8 under a windows-1251 header, and /header-1251 rendered as windows-1251 despite a utf-8 meta.

Is a missing charset a defect if the page is all ASCII?

Treat it as one. The page is one edit away from carrying a name with an accent, and the fallback is decided by the reader's browser, not by you. example.com declares nothing today and renders correctly only because every byte in it is below 128.

Can I read the encoding from the response body alone?

You can rule encodings out, not name one. A byte sequence that decodes under UTF-8 usually also decodes under windows-1251 with different letters. How to check the encoding of a file works through that asymmetry.

Verified

Verified by Maks Vernycurl 8.21.0Node 22.23.2Chrome 152.0.7977.76

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

basic7 minpublished updated Maks Verny