How to check for garbled characters from the wrong encoding

Run node encoding-report.mjs damaged.txt: it maps every character back to its windows-1252 byte and decodes those bytes as UTF-8. A report of cp1252 passes 1 means the text is mojibake and the original is recoverable. U+FFFD 2 means two bytes were destroyed and no repair exists.

Why check this

Two different accidents produce text a tester describes the same way, and only one of them can be fixed after the fact. A UTF-8 payload read by a component that assumed windows-1252 turns é into é: every byte survives, so the original text is still in the data. A windows-1252 file read as UTF-8 turns é into U+FFFD: the byte is gone, and every later export carries the hole.

Run this when an import, an export or a third-party feed changes, on the file as it arrives and again after your own pipeline has touched it. The failure it catches is an overnight job that writes repaired-looking rows into production while the source rows were already unrecoverable.

Prerequisites

• Node 22. TextDecoder follows the WHATWG Encoding Standard, which decides what happens to the five bytes windows-1252 leaves undefined.
• Python 3.13, which carries its own cp1252 codec and disagrees with Node on those bytes.
• Two samples you build yourself, so you know what the answer should be: one correct UTF-8 file, and one file written the way a windows-1252 application writes.

node -e "require('fs').writeFileSync('good.txt','José’s café costs 5 €','utf8')"
node -e "require('fs').writeFileSync('legacy.csv',Buffer.from('name,city\nJosé,Genève\n','latin1'))"
node -e "require('fs').writeFileSync('ru.txt','Ёлка','utf8')"

damage.mjs reproduces the fault on purpose: it reads UTF-8 bytes as windows-1252 and writes the result back out as UTF-8.

import { readFileSync, writeFileSync } from 'node:fs';
const text = new TextDecoder('windows-1252').decode(readFileSync(process.argv[2]));
writeFileSync(process.argv[3], text, 'utf8');
console.log(text);

encoding-report.mjs is the check. It builds the reverse of the windows-1252 table out of the decoder itself, so all 256 byte values are covered, and it repeats the undo until the undo stops working.

import { readFileSync } from 'node:fs';

const text = new TextDecoder('utf-8').decode(readFileSync(process.argv[2]));
const dec = new TextDecoder('windows-1252');
const rev = new Map();
for (let b = 0; b < 256; b += 1) rev.set(dec.decode(Uint8Array.of(b)), b);

function undo(s) {
  if ([...s].some((c) => !rev.has(c))) return null;
  try {
    return new TextDecoder('utf-8', { fatal: true })
      .decode(Uint8Array.from([...s], (c) => rev.get(c)));
  } catch { return null; }
}

let passes = 0;
let current = text;
for (let next = undo(current); next !== null; next = undo(current)) {
  current = next;
  passes += 1;
}

console.log('characters   ', [...text].length);
console.log('U+FFFD       ', [...text].filter((c) => c === '�').length);
console.log('cp1252 passes', passes);
console.log('recovered    ', passes > 0 ? current : '(nothing to undo)');

Steps

1. 1

   Step 1.

   Produce the damage yourself, so the shape of it is familiar before it arrives in a bug report.

   node damage.mjs good.txt damaged.txt
   

   José’s café costs 5 €

   é became é and ’ became ’. Those two pairs are the signature of UTF-8 read as windows-1252.

2. 2

   Step 2.

   Run the report on the damaged file.

   node encoding-report.mjs damaged.txt
   

   characters    27
   U+FFFD        0
   cp1252 passes 1
   recovered     José’s café costs 5 €

   21 characters became 27, because each accented letter turned into two. One undo pass recovers the original, so nothing was lost.

3. 3

   Step 3.

   Run the same report on the file that went the other way, decoded as UTF-8 when its bytes were windows-1252.

   node encoding-report.mjs legacy.csv
   

   characters    22
   U+FFFD        2
   cp1252 passes 0
   recovered     (nothing to undo)

   Bytes 0xE9 and 0xE8 were replaced by U+FFFD on decode. No undo exists, because those bytes are no longer in the string. This copy of the data is finished.

4. 4

   Step 4.

   Test for double encoding by damaging the damaged file a second time.

   node damage.mjs damaged.txt double.txt && node encoding-report.mjs double.txt
   

   José’s café costs 5 €
   characters    40
   U+FFFD        0
   cp1252 passes 2
   recovered     José’s café costs 5 €

   cp1252 passes 2 is the count of times the text went through the wrong decoder. A single-pass repair on this file leaves José’s behind and looks like a partial fix.

5. 5

   Step 5.

   Read code points rather than glyphs, on text whose bytes land in the gap of the windows-1252 table.

   node -e "const s=new TextDecoder('windows-1252').decode(require('fs').readFileSync(process.argv[1]));console.log('visible glyphs',[...s].filter(c=>c.codePointAt(0)>0x9f).length);console.log('code points   ',[...s].map(c=>'U+'+c.codePointAt(0).toString(16).toUpperCase().padStart(4,'0')).join(' '))" ru.txt
   

   visible glyphs 7
   code points    U+00D0 U+0081 U+00D0 U+00BB U+00D0 U+00BA U+00D0 U+00B0

   ru.txt holds the Russian word for fir tree. Its first byte pair is D0 81, and 0x81 has no character in windows-1252. Node maps it to U+0081, an invisible control. The damaged string is 8 characters wide and shows 7 glyphs, so a visual comparison undercounts it.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | cp1252 passes 1 | UTF-8 was read as windows-1252 once | Repair from the report, and fix the reader rather than the data. | | cp1252 passes 2 or more | The same text went through the wrong decoder twice | Repair the same number of times, and find both readers. | | U+FFFD above 0 | Bytes were replaced on decode | Go back to the source file. This copy cannot be repaired. | | Both counts 0 | No windows-1252 damage in this file | The complaint is about a font or a locale, not an encoding. | | More characters than the text should hold | Invisible C1 controls are in the string | Read the code points, as in step 5. |

Common mistakes

What to check next

• How to check if a page is UTF-8: the declaration a browser reads, and why a page with no charset falls back to windows-1252.
• How to check the encoding of a file: identifies what a file holds before anything decodes it.
• How to check the encoding of a CSV file: the spreadsheet path, where a BOM decides what the reader assumes.
• How to test unicode input: catches the same fault at the form, before it reaches storage.
• Localization testing checklist: where this check sits in a release pass.

FAQ

How to check if text is double encoded utf-8?

Undo the windows-1252 step repeatedly and count how many passes succeed. The report in step 4 prints cp1252 passes 2 for text that went through the wrong decoder twice. Text damaged once prints 1, clean text prints 0.

Is mojibake always reversible?

When every byte survived, yes. When the decoder replaced bytes with U+FFFD, no. The two cases read alike in a bug report, and the report separates them in one run.

Why do accented letters show as two characters and Cyrillic as one plus a gap?

Both expand to two characters. Cyrillic bytes often land in the 0x80 to 0x9F range, which windows-1252 either leaves undefined or maps to an invisible control, so only one character of the pair is visible.

Can I find this without writing a script?

Not reliably. A search for Ã finds the common Latin cases and misses every string whose second byte is a C1 control. The undo test covers both, and it also tells you how many passes were involved.

Verified