h2check

How to test unicode input

Serve a form from a local server, type a Cyrillic, combining mark and emoji value into it with Chrome, submit, and read the stored value back as code points. The round trip should be identical. A field with maxlength stored 4 of 11 code units and dropped an accent.

Why check this

Input handling breaks in places a screenshot cannot show. The value looks right in the field and wrong in the database, or right in the database and wrong after an export. Reading code points on both ends is the only comparison that survives a font substitution and a terminal that cannot render the character.

Run this when a form gains a length limit, when a field becomes searchable, and before any release that opens sign-up to a new market. The failure it catches is a user whose display name is stored with the accent removed, who then cannot find their own record, because the search term still carries the accent.

Prerequisites

import { createServer } from 'node:http';

const FORM = `<!doctype html><html lang="en"><meta charset="utf-8"><title>input</title>
<form method="post" action="/save" accept-charset="utf-8">
<label>full <input name="full" id="full" size="40"></label>
<label>emoji <input name="emoji" id="emoji" maxlength="5"></label>
<label>cut <input name="cut" id="cut" maxlength="4"></label>
<label>accent <input name="accent" id="accent" maxlength="4"></label>
<button id="send">Save</button></form>`;

let stored = {};

createServer((req, res) => {
  if (req.method === 'POST') {
    let body = '';
    req.on('data', (c) => { body += c; });
    req.on('end', () => {
      stored = Object.fromEntries(new URLSearchParams(body));
      res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
      res.end('<!doctype html><html lang="en"><meta charset="utf-8"><p id="ok">saved</p>');
    });
    return;
  }
  if (req.url === '/read') {
    res.writeHead(200, { 'content-type': 'application/json; charset=utf-8' });
    res.end(JSON.stringify(stored));
    return;
  }
  res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
  res.end(FORM);
}).listen(8973, '127.0.0.1', () => console.log('listening on 127.0.0.1:8973'));

// drive.mjs
import { launch } from 'puppeteer-core';
const CHROME = process.env.CHROME ?? 'C:/Program Files/Google/Chrome/Application/chrome.exe';

const family = String.fromCodePoint(0x1f468, 0x200d, 0x1f469, 0x200d, 0x1f467, 0x200d, 0x1f466);
const accent = String.fromCodePoint(99, 97, 102, 101, 0x301);
const full = 'Київ ' + accent + ' ' + family;
const points = (s) => [...s].map((c) => 'U+' + c.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')).join(' ');
const lone = (s) => /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/.test(s);

const browser = await launch({ executablePath: CHROME, headless: true });
const page = await browser.newPage();
await page.goto('http://127.0.0.1:8973/', { waitUntil: 'networkidle2' });
await page.type('#full', full);
await page.type('#emoji', family);
await page.type('#cut', family);
await page.type('#accent', accent);
await Promise.all([page.click('#send'), page.waitForNavigation()]);
const typed = await page.evaluate(() => fetch('/read').then((r) => r.json()));
console.log('full   typed', full.length, 'units, stored', typed.full.length, 'identical', typed.full === full);
console.log('emoji  maxlength=5, typed', family.length, 'units, stored', typed.emoji.length);
console.log('      ', points(typed.emoji), 'lone surrogate', lone(typed.emoji));
console.log('cut    maxlength=4, typed', family.length, 'units, stored', typed.cut.length);
console.log('      ', points(typed.cut), 'prefix of input', family.startsWith(typed.cut));
console.log('accent maxlength=4, typed', accent.length, 'units, stored', typed.accent.length);
console.log('      ', points(typed.accent), JSON.stringify(typed.accent));

await page.goto('http://127.0.0.1:8973/', { waitUntil: 'networkidle2' });
await page.evaluate((v) => { document.querySelector('#cut').value = v; }, family);
await Promise.all([page.click('#send'), page.waitForNavigation()]);
const scripted = await page.evaluate(() => fetch('/read').then((r) => r.json()));
console.log('cut    maxlength=4, value set by script, stored', scripted.cut.length, 'units');
console.log('chrome', await browser.version());
await browser.close();

const nfc = String.fromCodePoint(99, 97, 102, 0xe9);
const nfd = String.fromCodePoint(99, 97, 102, 101, 0x301);
console.log('on screen      ', nfc, nfd);
console.log('raw equal      ', nfc === nfd);
console.log('code units     ', nfc.length, nfd.length);
console.log('NFC equal      ', nfc.normalize('NFC') === nfd.normalize('NFC'));
console.log('NFC code units ', nfc.normalize('NFC').length, nfd.normalize('NFC').length);

Steps

  1. Step 1.

    Confirm the target is up and declares its charset, so a later fault cannot be blamed on the response header.

    curl -sI http://127.0.0.1:8973/ | grep -i "^content-type" && curl -s http://127.0.0.1:8973/ | head -2

    content-type: text/html; charset=utf-8
<!doctype html><html lang="en"><meta charset="utf-8"><title>input</title>
<form method="post" action="/save" accept-charset="utf-8">

    The header and the meta element agree, and the form declares accept-charset. Anything wrong from here is the field or the store, not the encoding of the page.

  2. Step 2.

    Type the values into the form with a browser, submit, and read the stored result back as code points.

    node drive.mjs

    full   typed 22 units, stored 22 identical true
emoji  maxlength=5, typed 11 units, stored 5
     U+1F468 U+200D U+1F469 lone surrogate false
cut    maxlength=4, typed 11 units, stored 4
     U+1F468 U+200D U+200D prefix of input false
accent maxlength=4, typed 5 units, stored 4
     U+0063 U+0061 U+0066 U+0065 "cafe"
cut    maxlength=4, value set by script, stored 11 units
chrome Chrome/152.0.7977.76

    Read the cut line twice. The stored value is U+1F468 U+200D U+200D, two joiners in a row, and prefix of input false says it is not the first four units of what was typed. The limit is applied per keystroke, so characters that do not fit are dropped while later, shorter ones still get in.

  3. Step 3.

    Insert both spellings of one word into a column with a unique constraint.

    sqlite3 :memory: "create table users(name text unique); insert into users values (char(99,97,102,233)); insert into users values (char(99,97,102,101,769)); select rowid, name, length(name) as chars, length(cast(name as blob)) as bytes from users; select 'equal in sql: ' || (char(99,97,102,233) = char(99,97,102,101,769));"

    1|café|4|5
2|café|5|6
equal in sql: 0

    Two rows, the same word on screen, and the unique index raised nothing. One is a precomposed U+00E9, the other is e plus a combining acute. Comparison in SQL is over bytes, so they are different keys.

  4. Step 4.

    Normalise both spellings and compare again.

    node normalise.mjs

    on screen       café café
raw equal       false
code units      4 5
NFC equal       true
NFC code units  4 4

    NFC collapses the combining sequence into the precomposed code point, and the two values become one key. Normalise on the way in, before validation and before the insert, and the unique constraint starts working.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | identical true on the unlimited field | The round trip preserved every code point | The transport is fine. Look at the limited fields. | | prefix of input false | The limit dropped characters from the middle | State the limit in graphemes and enforce it after typing, not during. | | A stored value ending in U+200D | A joiner survived the character it joined | Strip trailing joiners and combining marks after any truncation. | | "cafe" where café was typed | A combining mark fell outside the limit | Normalise to NFC before the length is measured. | | Two rows that look identical in a unique column | The values differ in normalisation | Normalise on write, then deduplicate the existing rows. |

Common mistakes

Sign: A maxlength field stores characters that were never next to each other in the input.Cause: Chrome 152 applies maxlength while the value is edited, not once at the end. Typing a family emoji into maxlength=4 stored U+1F468 U+200D U+200D: the second person did not fit, the joiner after it did, and the result is not a prefix of what was typed. A test that compares the first four units of the input to the stored value passes on the wrong data.
Sign: maxlength is treated as a guarantee about what reaches the server.Cause: It constrains user edits only. In the same run, assigning the full 11-unit value to the element from a script and submitting stored all 11 units. Any limit that matters has to be checked again on the server.
Sign: A unique constraint allows two accounts with the same name.Cause: SQLite, and any store comparing bytes, treats café written as four code points and café written as five as different keys. Neither the form nor the index objects. Normalise to NFC before the insert, or the duplicate is created by a user switching operating systems.
Sign: Testing by posting a body with curl finds none of this.Cause: A request built by hand never meets the input element, so no attribute on the field is exercised. Type into the form with a browser, then read the stored value, or the test only covers the part that already worked.

What to check next

FAQ

How to test emoji input?

Type a zero width joiner sequence, a flag and a skin tone variant into every limited field, then read the stored value as code points. A count alone hides the fault. The sequence stored in step 2 is 4 code units long and holds two adjacent joiners.

Which characters belong in a test value?

One Cyrillic or Greek word, one combining mark, one astral character and one right-to-left word. That set covers multi-byte encoding, normalisation, surrogate pairs and direction in a single field.

Should the application normalise input?

Normalise to NFC before validation and before storage, and compare normalised values. Keep the original only where the difference carries meaning, which is rare outside linguistic tools.

Why type into the form instead of posting the body?

The input element and its attributes only run during typing. The maxlength behaviour in step 2 does not happen for a POST built by a script, and neither does an input mask or a paste handler.

Can this run against a deployed site?

Yes, once the value is yours to store. Use a local target for the first pass, because it isolates the field and the store from a proxy, and a failure there is unambiguous.

Verified

Verified by Maks Vernynode 22.23.2Chrome 152.0.7977.76puppeteer-core 25.10.0sqlite3 3.50.6curl 8.1.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

intermediate10 minpublished updated Maks Verny