How to check the encoding of a file

Nothing can name the encoding of a file with certainty, so ask a different question: is it valid as UTF-8? Run iconv -f UTF-8 -t UTF-8 file and read the exit code, then decode the bytes under each candidate with Node's TextDecoder in fatal mode and compare what each one produced.

Why check this

An encoding is not stored in the file. A text file is bytes plus an assumption, and the assumption travels in a header, a database column, a parser setting or somebody's memory. Run this check whenever a file crosses a boundary: an export from one system imported into another, a fixture committed by a colleague on a different operating system, a translation file returned by an agency.

The failure it prevents is silent corruption. A windows-1251 export loaded as windows-1252 produces text that is structurally intact, passes every schema check, and turns every Cyrillic name into Latin punctuation. Nothing throws.

Prerequisites

• Node 22 for TextDecoder. The TextDecoder options reference covers the fatal flag used below.
• iconv and file, both in Git Bash on Windows and in any Linux shell.
• Six fixtures with known contents, so you can tell a correct answer from a plausible one.

printf 'Zurich cafe, Привет\n' > utf8.txt
printf '\xef\xbb\xbf' > utf8-bom.txt && cat utf8.txt >> utf8-bom.txt
iconv -f UTF-8 -t UTF-16LE utf8.txt > utf16le-nobom.txt
printf '\xff\xfe' > utf16le.txt && cat utf16le-nobom.txt >> utf16le.txt
iconv -f UTF-8 -t WINDOWS-1251 utf8.txt > cp1251.txt
printf 'Caf\xe9, na\xefve\n' > latin1.txt

• sniff.mjs, which decodes one file under four candidate encodings and reports each result.

import { readFileSync } from 'node:fs';
const buf = readFileSync(process.argv[2]);
const bom =
  buf[0] === 0xef && buf[1] === 0xbb && buf[2] === 0xbf ? 'utf-8'
  : buf[0] === 0xff && buf[1] === 0xfe ? 'utf-16le'
  : buf[0] === 0xfe && buf[1] === 0xff ? 'utf-16be'
  : 'none';
console.log(`${process.argv[2]}  ${buf.length} bytes  BOM: ${bom}`);
for (const enc of ['utf-8', 'utf-16le', 'windows-1251', 'windows-1252']) {
  try {
    const text = new TextDecoder(enc, { fatal: true }).decode(buf);
    console.log(`  ${enc.padEnd(13)} valid    ${text.trim().replace(/\s+/g, ' ').slice(0, 40)}`);
  } catch (e) {
    console.log(`  ${enc.padEnd(13)} INVALID  ${e.message}`);
  }
}

Steps

1. 1

   Step 1.

   Read the first four bytes of every file. A BOM is the only self-declaration a text file has.

   for f in *.txt; do printf '%-18s %s\n' "$f" "$(head -c 4 "$f" | xxd -p)"; done
   

   cp1251.txt         5a757269
   latin1.txt         436166e9
   utf16le-nobom.txt  5a007500
   utf16le.txt        fffe5a00
   utf8-bom.txt       efbbbf5a
   utf8.txt           5a757269

   efbbbf is a UTF-8 BOM, fffe a UTF-16LE BOM. The other four files declare nothing.

2. 2

   Step 2.

   Ask whether the bytes are valid UTF-8. iconv from UTF-8 to UTF-8 is a validity gate: exit 0 means every sequence decoded.

   for f in *.txt; do printf '%-18s ' "$f"; iconv -f UTF-8 -t UTF-8 "$f" > /dev/null 2>&1; echo "exit $?"; done
   

   cp1251.txt         exit 1
   latin1.txt         exit 1
   utf16le-nobom.txt  exit 0
   utf16le.txt        exit 1
   utf8-bom.txt       exit 0
   utf8.txt           exit 0

   Three exit 1 results are real answers: those files are not UTF-8. The exit 0 on utf16le-nobom.txt is the trap in step 4.

3. 3

   Step 3.

   Decode each file under every candidate and read what came out.

   for f in utf8.txt utf8-bom.txt utf16le.txt cp1251.txt; do node sniff.mjs "$f"; done
   

   utf8.txt  26 bytes  BOM: none
   utf-8         valid    Zurich cafe, Привет
   utf-16le      valid    畚楲档挠晡ⱥ퀠톟킀킸킲통ં
   windows-1251  valid    Zurich cafe, Привет
   windows-1252  valid    Zurich cafe, Привет
   utf8-bom.txt  29 bytes  BOM: utf-8
   utf-8         valid    Zurich cafe, Привет
   utf-16le      INVALID  The encoded data was not valid for encoding utf-16le
   windows-1251  valid    Zurich cafe, Привет
   windows-1252  valid    Zurich cafe, Привет
   utf16le.txt  42 bytes  BOM: utf-16le
   utf-8         INVALID  The encoded data was not valid for encoding utf-8
   utf-16le      valid    Zurich cafe, Привет
   windows-1251  valid    яюZ u r i c h   c a f e ,   @825B
   windows-1252  valid    ÿþZ u r i c h   c a f e ,   @825B
   cp1251.txt  20 bytes  BOM: none
   utf-8         INVALID  The encoded data was not valid for encoding utf-8
   utf-16le      valid    畚楲档挠晡ⱥ켠૲
   windows-1251  valid    Zurich cafe, Привет
   windows-1252  valid    Zurich cafe, Ïðèâåò

   cp1251.txt decodes without error under two encodings and means two different things. No tool can choose between them from the bytes.

4. 4

   Step 4.

   Compare that with the answer file gives, which is the answer most people quote.

   file -i *.txt
   

   cp1251.txt:        text/plain; charset=iso-8859-1
   latin1.txt:        text/plain; charset=iso-8859-1
   utf16le-nobom.txt: application/octet-stream; charset=binary
   utf16le.txt:       text/plain; charset=utf-16le
   utf8-bom.txt:      text/plain; charset=utf-8
   utf8.txt:          text/plain; charset=utf-8

   cp1251.txt is windows-1251 and file calls it iso-8859-1, with no hedge. It tests byte ranges, and the two encodings occupy the same range.

5. 5

   Step 5.

   Read the file the way the program that consumes it will read it. On Windows that is rarely UTF-8 by default.

   [System.Text.Encoding]::Default.WebName
   (Get-Content ./utf8.txt -Raw).Trim()
   (Get-Content ./utf8.txt -Raw -Encoding utf8).Trim()
   (Get-Content ./utf8-bom.txt -Raw).Trim()
   

   windows-1251
   Zurich cafe, Привет
   Zurich cafe, Привет
   Zurich cafe, Привет

   Same file, three readings. Without -Encoding the cmdlet used the machine ANSI code page. The BOM made the third reading correct without any flag.

6. 6

   Step 6.

   Convert to UTF-8 and run the gate again, so the fix is verified rather than assumed.

   iconv -f WINDOWS-1251 -t UTF-8 cp1251.txt > fixed.txt && iconv -f UTF-8 -t UTF-8 fixed.txt > /dev/null; echo "gate exit $?"; cat fixed.txt
   

   gate exit 0
   Zurich cafe, Привет

   Exit 0 and readable text. Either one alone would not be enough.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | The UTF-8 gate exits 1 | The file is definitely not UTF-8 | Run sniff.mjs and pick the candidate whose text is readable in the expected language. | | The gate exits 0 | The bytes could be UTF-8 | Confirm with the content. Pure ASCII and UTF-16 of ASCII text both pass. | | Two candidates decode without error | The bytes are ambiguous | Decide from the source system or the language of the text, not from a tool. | | file names an encoding | A guess from byte ranges | Treat it as a hint. It reported iso-8859-1 for a windows-1251 file above. |

Common mistakes

What to check next

• How to check the encoding of a CSV file: the same bytes with a parser on top, and what a BOM does to a header row.
• How to check JSON encoding: why JSON narrows this question to one legal answer.
• How to check if a page is UTF-8: the same file served over HTTP, where a header can override it.
• How to check for garbled characters from the wrong encoding: reading the damage backwards to name the pair of encodings involved.
• How to test unicode input: the same failure through a form rather than a file.

FAQ

How do I check if a file is UTF-8 from the command line?

iconv -f UTF-8 -t UTF-8 file > /dev/null; echo $?. Exit 1 means it is not. Exit 0 means it could be, and pure ASCII files always pass, as does UTF-16 of Latin text.

How do I check the encoding of a string rather than a file?

A string in a running program has no encoding; it is already decoded characters. Print the code points, for example [...s].map((c) => c.codePointAt(0).toString(16)). Mojibake shows up as a run of Latin-1 code points where letters should be.

Does the BOM decide the encoding?

It decides it for the tools that look for one, which is most editors and Windows PowerShell. It is three bytes of content for every tool that does not, which is why it breaks parsers in the CSV and JSON pages.

Why do two encodings both decode the same file?

Single-byte encodings map all 256 byte values, so no byte can be invalid in them. cp1251.txt decoded under windows-1251 and windows-1252 in step 3, correctly in one and as Latin punctuation in the other.

Verified