How to test a unicode file name in content disposition

Upload eight files whose names carry Cyrillic, CJK, an emoji, a combining accent and a right-to-left override, then compare the filename bytes in the part header against the name your parser produced and the name on disk. All three should agree. Here busboy read every non-ASCII name as Latin-1.

Why check this

Run this when an upload route is added, when its parser changes version, and at staging sign-off for a product that takes files in more than one language. The name passes four layers that each guess an encoding: client, parser, storage code, filesystem.

The failure it prevents is a file that uploads with a 200 and cannot be found afterwards. A user sends звіт.csv, the response says success, and the admin list shows eight Latin-1 characters where four letters were. The report arrives weeks later, as a search problem.

A quieter one: two uploads whose names look identical land as two rows, because one accent is precomposed and the other a combining mark.

Prerequisites

import { createServer } from 'node:http';
import { mkdirSync, writeFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';
import busboy from 'busboy';

const STORE = new URL('./store/', import.meta.url).pathname.slice(1);
mkdirSync(STORE, { recursive: true });

const hex = (c) => c.codePointAt(0).toString(16).toUpperCase().padStart(4, '0');
const pts = (s) => [...s].map((c) => 'U+' + hex(c)).join(' ');
// Format characters (Unicode Cf) reorder or vanish in a terminal and in HTML, so
// print them escaped. U+202E in a name flips everything after it on screen.
const show = (s) => JSON.stringify(s).replace(/\p{Cf}/gu, (c) => '\\u' + hex(c));

createServer((req, res) => {
  const send = (t) => { res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8' }); res.end(t); };

  if (req.url === '/list') {
    send(readdirSync(STORE).map((n) => `${statSync(join(STORE, n)).size} B  ${pts(n)}`).join('\n') + '\n');
    return;
  }
  if (req.url === '/raw') {                       // the part header exactly as it arrived
    const c = [];
    req.on('data', (d) => c.push(d)).on('end', () => send(
      Buffer.concat(c).toString('latin1').split('\r\n')
        .filter((l) => /^content-disposition/i.test(l))
        .map((l) => l.replace(/[^\x20-\x7e]/g, (ch) => '\\x' + ch.charCodeAt(0).toString(16)))
        .join('\n') + '\n'));
    return;
  }
  if (req.url === '/upload') {
    const bb = busboy({ headers: req.headers });
    const out = [];
    bb.on('file', (field, stream, info) => {
      const raw = info.filename;                             // what busboy 1.6.0 hands you
      const utf8 = Buffer.from(raw, 'latin1').toString('utf8');
      const body = [];
      stream.on('data', (d) => body.push(d)).on('end', () => {
        out.push(`raw    ${pts(raw)}`);
        out.push(`utf8   ${show(utf8)}  ${pts(utf8)}`);
        out.push(`nfc    ${utf8 === utf8.normalize('NFC')}`);
        try { writeFileSync(join(STORE, utf8), Buffer.concat(body)); out.push('disk   ok'); }
        catch (e) { out.push(`disk   ${e.code}`); }
      });
    });
    bb.on('close', () => send(out.join('\n') + '\n'));
    req.pipe(bb);
    return;
  }
  res.writeHead(404).end();
}).listen(8935, '127.0.0.1', () => console.log('listening on 127.0.0.1:8935'));
const url = process.argv[2] ?? 'http://127.0.0.1:8935/upload';
const cp = (...n) => String.fromCodePoint(...n);

const names = {
  ascii: 'report.csv',
  cyrillic: cp(0x437, 0x432, 0x456, 0x442) + '.csv',   // zvit, Ukrainian
  cjk: cp(0x5831, 0x544a, 0x66f8) + '.csv',            // houkokusho, Japanese
  emoji: 'report' + cp(0x1f4c4) + '.csv',
  nfc: 'caf' + cp(0xe9) + '.csv',                      // e with acute, one code point
  nfd: 'cafe' + cp(0x301) + '.csv',                    // e plus a combining acute
  rtl: 'annex' + cp(0x202e) + 'fdp.exe',               // right-to-left override
  colon: 'q1' + cp(0x3a) + 'q2.csv',                   // the colon that NTFS reserves
};

const body = new TextEncoder().encode('id,amount\n1,42\n');

for (const [label, name] of Object.entries(names)) {
  const fd = new FormData();
  fd.set('file', new File([body], name, { type: 'text/csv' }));
  const res = await fetch(url, { method: 'POST', body: fd });
  console.log(`=== ${label}`);
  process.stdout.write(await res.text());
}

Steps

  1. Step 1.

    Send one Cyrillic name with curl and read the bytes that left the client.

    curl -s -F 'file=@payload.csv;filename=звіт.csv' http://127.0.0.1:8935/raw
    
    Content-Disposition: form-data; name="file"; filename="\xe7\xe2\xb3\xf2.csv"

    Four bytes for four letters, so this is not UTF-8. e7 e2 b3 f2 is звіт in Windows-1251, the ANSI code page of this machine.

  2. Step 2.

    Send all eight names from a browser-shaped client and read the part headers again.

    node upload.mjs http://127.0.0.1:8935/raw
    
    === ascii
    Content-Disposition: form-data; name="file"; filename="report.csv"
    === cyrillic
    Content-Disposition: form-data; name="file"; filename="\xd0\xb7\xd0\xb2\xd1\x96\xd1\x82.csv"
    === cjk
    Content-Disposition: form-data; name="file"; filename="\xe5\xa0\xb1\xe5\x91\x8a\xe6\x9b\xb8.csv"
    === emoji
    Content-Disposition: form-data; name="file"; filename="report\xf0\x9f\x93\x84.csv"
    === nfc
    Content-Disposition: form-data; name="file"; filename="caf\xc3\xa9.csv"
    === nfd
    Content-Disposition: form-data; name="file"; filename="cafe\xcc\x81.csv"
    === rtl
    Content-Disposition: form-data; name="file"; filename="annex\xe2\x80\xaefdp.exe"
    === colon
    Content-Disposition: form-data; name="file"; filename="q1:q2.csv"

    Eight bytes for four Cyrillic letters, three per CJK ideograph, four for the emoji: raw UTF-8 in a quoted string, with no charset label in the part. nfc and nfd differ on the wire, c3 a9 against 65 cc 81.

  3. Step 3.

    Parse the same eight uploads and read the name the application receives.

    node upload.mjs http://127.0.0.1:8935/upload
    
    …
    === cyrillic
    raw    U+00D0 U+00B7 U+00D0 U+00B2 U+00D1 U+0096 U+00D1 U+0082 U+002E U+0063 U+0073 U+0076
    utf8   "звіт.csv"  U+0437 U+0432 U+0456 U+0442 U+002E U+0063 U+0073 U+0076
    nfc    true
    disk   ok
    …
    === emoji
    raw    U+0072 U+0065 U+0070 U+006F U+0072 U+0074 U+00F0 U+009F U+0093 U+0084 U+002E U+0063 U+0073 U+0076
    utf8   "report📄.csv"  U+0072 U+0065 U+0070 U+006F U+0072 U+0074 U+1F4C4 U+002E U+0063 U+0073 U+0076
    nfc    true
    disk   ok
    === nfc
    raw    U+0063 U+0061 U+0066 U+00C3 U+00A9 U+002E U+0063 U+0073 U+0076
    utf8   "café.csv"  U+0063 U+0061 U+0066 U+00E9 U+002E U+0063 U+0073 U+0076
    nfc    true
    disk   ok
    === nfd
    raw    U+0063 U+0061 U+0066 U+0065 U+00CC U+0081 U+002E U+0063 U+0073 U+0076
    utf8   "café.csv"  U+0063 U+0061 U+0066 U+0065 U+0301 U+002E U+0063 U+0073 U+0076
    nfc    false
    disk   ok
    === rtl
    raw    U+0061 U+006E U+006E U+0065 U+0078 U+00E2 U+0080 U+00AE U+0066 U+0064 U+0070 U+002E U+0065 U+0078 U+0065
    utf8   "annex\u202Efdp.exe"  U+0061 U+006E U+006E U+0065 U+0078 U+202E U+0066 U+0064 U+0070 U+002E U+0065 U+0078 U+0065
    nfc    true
    disk   ok
    …

    Every raw code point sits in U+0080 to U+00FF: the UTF-8 bytes read one at a time as Latin-1. The utf8 line re-decodes them into the real name. nfd prints café.csv exactly as nfc does, and nfc false is the only signal that the two differ.

  4. Step 4.

    Read the names that reached the filesystem, as code points and sizes.

    curl -s http://127.0.0.1:8935/list
    
    15 B  U+0061 U+006E U+006E U+0065 U+0078 U+202E U+0066 U+0064 U+0070 U+002E U+0065 U+0078 U+0065
    15 B  U+0063 U+0061 U+0066 U+0065 U+0301 U+002E U+0063 U+0073 U+0076
    15 B  U+0063 U+0061 U+0066 U+00E9 U+002E U+0063 U+0073 U+0076
    0 B  U+0071 U+0031
    15 B  U+0072 U+0065 U+0070 U+006F U+0072 U+0074 U+002E U+0063 U+0073 U+0076
    15 B  U+0072 U+0065 U+0070 U+006F U+0072 U+0074 U+1F4C4 U+002E U+0063 U+0073 U+0076
    15 B  U+0437 U+0432 U+0456 U+0442 U+002E U+0063 U+0073 U+0076
    15 B  U+5831 U+544A U+66F8 U+002E U+0063 U+0073 U+0076

    Eight uploads, eight entries, two of them the accented word that NTFS kept apart. One entry is a name nobody sent: q1, holding zero bytes.

  5. Step 5.

    Find the 15 bytes that the q1:q2.csv upload reported as written.

    Get-Item -Path .\store\q1 -Stream * | Format-Table Stream,Length -AutoSize
    
    Stream Length
    ------ ------
    :$DATA      0
    q2.csv     15

    The colon read as a stream separator, so the body went into an NTFS alternate data stream on a zero-length file. writeFileSync raised nothing and the route logged success.

  6. Step 6.

    Confirm the finding on a server you do not control, against an ASCII control.

    for n in report.csv звіт.csv; do printf '%-12s ' "$n"; curl -s -o resp.json -w "HTTP %{http_code}  " -F "file=@payload.csv;filename=$n" https://httpbin.org/post; node -e "const j=require('./resp.json');console.log('files',JSON.stringify(Object.keys(j.files)),' form',JSON.stringify(Object.keys(j.form)))"; done
    
    report.csv   HTTP 200  files ["file"]  form []
    звіт.csv HTTP 200  files []  form []

    Same endpoint, same body, one character class apart. The Windows-1251 name gives 200 OK with no file and no field, because httpbin's parser dropped the part in silence. That is the response a user gets when an upload vanishes.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Two bytes per Cyrillic letter in /raw | The client sent UTF-8, which is what browsers do | Nothing. Move to what the parser made of it. | | One byte per letter, e7 e2 b3 f2 | The client sent a legacy code page, here Windows-1251 | Test with a browser or a scripted UTF-8 client. A desktop curl is not the reference. | | Parser name is all U+0080 to U+00FF | The parser decoded UTF-8 bytes as Latin-1 | Re-decode with Buffer.from(name, 'latin1').toString('utf8') before any validation. | | U+FFFD in the re-decoded name | The bytes were not UTF-8 to begin with | Reject the upload. Guessing the code page produces a name nobody can reproduce. | | nfc false | The name carries a combining mark, not a precomposed character | Normalise to NFC before the duplicate check and before the write. | | Two stored names that look the same | The store keeps both spellings as separate keys | Normalise on write, then deduplicate what is already there. | | A stored file at 0 B after a successful write | A character in the name has meaning to the filesystem | Reject or replace the character. On NTFS the colon hides the body in a stream. | | U+202E or another Cf code point in the name | The displayed name is not the stored name | Strip format characters, and escape them wherever the name is shown. |

Common mistakes

Sign: Non-ASCII names arrive as two Latin-1 characters per letter, and the fix is applied in the template rather than at the parser.Cause: busboy 1.6.0 decodes the filename parameter as Latin-1. The four Cyrillic letters arrived as U+00D0 U+00B7 U+00D0 U+00B2 U+00D1 U+0096 U+00D1 U+0082, eight code points instead of four. Every length check, extension check and duplicate check downstream then runs on the wrong string, so re-decoding only at display time leaves all of them broken.
Sign: The re-decode is applied everywhere and starts producing U+FFFD on some uploads.Cause: Buffer.from(name, 'latin1').toString('utf8') is correct only when the client sent UTF-8. The curl run in step 1 sent Windows-1251, and the same re-decode turned 4 bytes into 3 replacement characters. RFC 7578 gives the part no charset parameter, so a server cannot know. Treat an undecodable name as an invalid upload rather than repairing it.
Sign: A duplicate-name check passes and the user ends up with two files that read alike.Cause: One name spells the accent as U+00E9 and the other as e followed by U+0301. They are different byte strings, so the comparison, the unique index and NTFS all keep both. Only the nfc false line in step 3 distinguishes them, and nothing in the upload response does.
Sign: An upload is stored, the route logs success, and the file has zero bytes.Cause: Microsoft's Naming Files, Paths, and Namespaces reserves the colon and reads name:stream as an alternate data stream, which is what happened here: writeFileSync threw nothing and put the 15-byte body into a stream on a zero-length file. Any test that asserts only that the write did not throw passes on this. Assert the stored size instead.

What to check next

FAQ

Can I put filename* in a multipart upload part?

RFC 7578 section 4.2 forbids the RFC 5987 encoding in a form part, and browsers do not send it. Send raw UTF-8 bytes in filename, as every client in step 2 did.

Which test names cover the most ground?

The eight in upload.mjs: a two-byte script, a three-byte script, an astral character, two spellings of one accented word, a right-to-left override, and a name whose colon NTFS reserves.

Why build the names from code points instead of typing them?

A shell, an editor or a paste can change them first. The step 1 command was typed as UTF-8 and left as Windows-1251.

Should the server rename uploads instead?

A generated id with the original name in metadata removes the filesystem questions. The decoding and normalisation ones stay, because the metadata still holds a name.

Does the emoji need its own case?

Yes. It is the only name above U+FFFF, so it separates a code-point count from a UTF-16 code-unit count. It arrived as 4 bytes and 1 code point, with length 2.

Verified

Verified by Maks Vernycurl 8.21.0node 22.23.2busboy 1.6.0Windows PowerShell 5.1

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

intermediate8 minpublished updated Maks Verny