Autofill testing

Walk every control and check its autocomplete value against the 54 field names in the HTML standard. node probe.mjs audit printed 19 controls here and 8 of them not ok. Then read document.querySelectorAll('input:autofill').length for what the browser filled. Typing and el.value never set that pseudo-class.

Why check this

The autocomplete attributes on a form are read by software nobody on the team wrote, and a defect in them survives every manual session. The browser fills from its own heuristics: in step 5 Chrome filled four card fields whose tokens were invented. A tester watches a form work over attributes that state nothing.

Run the audit when a form is added, when a field changes type, and before a checkout or sign-up release. One failure it prevents is the autocomplete="off" in step 4: four fields carried it, Chrome filled all four anyway, and the body they submitted matched the run with correct tokens. A ticket closed on off is closed on nothing.

Prerequisites

server.mjs, one account form with mixed attributes and a card form in three variants:

// server.mjs : one account form with a mixed set of autocomplete attributes,
// plus a card form served in three variants. Node 22, port 8934.
import { createServer } from 'node:http';

const PORT = 8934;

const FIELDS = [
  ['fullname', 'fullname', 'text', 'Full name', 'name'],
  ['email', 'email', 'email', 'Email', 'email'],
  ['phone', 'phone', 'tel', 'Phone', 'tel-number'],
  ['country', 'country', 'select', 'Country', 'country'],
  ['city', 'city', 'text', 'City', 'address-line1'],
  ['zip', 'zip', 'number', 'ZIP', 'shipping postal-code'],
  ['street', 'street', 'textarea', 'Street', 'shipping street-address'],
  ['billzip', 'billzip', 'text', 'Billing ZIP', 'section-billing billing postal-code'],
  ['account', 'account', 'text', 'Username', 'username1'],
  ['q', 'sitesearch', 'search', 'Search', ''],
  ['newpass', 'newpass', 'password', 'New password', 'new-password'],
  ['newpass2', 'confirm', 'password', 'Repeat password', 'new-password'],
  ['oldpass', 'oldpass', 'password', 'Current password', 'current-password'],
  ['otp', 'otp', 'text', 'SMS code', 'one-time-code'],
  ['cvc', 'cvc', 'text', 'Security code', 'off'],
  ['promo', 'promo', 'text', 'Promo code', null],
  ['news', 'news', 'checkbox', 'Email me', 'email'],
  ['worktel', 'worktel', 'tel', 'Work phone', 'work tel'],
  ['lastname', 'lastname', 'text', 'Last name', 'work family-name'],
];

const attr = (ac) => (ac === null ? '' : ' autocomplete="' + ac + '"');

const control = ([name, id, type, label, ac]) => {
  const common = 'id="' + id + '" name="' + name + '"' + attr(ac);
  if (type === 'select') return '<select ' + common + '><option>Ukraine</option></select>';
  if (type === 'textarea') return '<textarea ' + common + '></textarea>';
  return '<input ' + common + ' type="' + type + '">';
};

const accountPage = () => '<!doctype html><html lang="en"><meta charset="utf-8"><title>account</title>'
  + '<form id="f" method="post" action="/save">'
  + FIELDS.map((f) => '<p><label for="' + f[1] + '">' + f[3] + '</label> ' + control(f) + '</p>').join('')
  + '<button id="go">Save</button></form>';

const CARD = [
  ['ccname', 'Name on card', 'cc-name', 'cardholder'],
  ['ccnum', 'Card number', 'cc-number', 'card-number'],
  ['ccexp', 'Expiry', 'cc-exp', 'cc-expiry'],
  ['cvc', 'Security code', 'cc-csc', 'cvc2'],
];

const cardPage = (variant) => '<!doctype html><html lang="en"><meta charset="utf-8"><title>payment</title>'
  + '<form id="pay" method="post" action="/pay">'
  + CARD.map(([id, label, good, bogus]) => {
    const ac = variant === 'off' ? 'off' : variant === 'bogus' ? bogus : good;
    return '<p><label for="' + id + '">' + label + '</label> <input id="' + id + '" name="' + id + '" autocomplete="' + ac + '"></p>';
  }).join('')
  + '<button id="pay-go">Pay</button></form>';

createServer((req, res) => {
  const url = new URL(req.url, 'http://127.0.0.1');
  if (req.method === 'POST') {
    const chunks = [];
    req.on('data', (c) => chunks.push(c));
    req.on('end', () => {
      const body = Buffer.concat(chunks).toString();
      console.log('POST ' + url.pathname + ' body: ' + body);
      res.writeHead(200, { 'content-type': 'text/html; charset=utf-8', 'cache-control': 'no-store' });
      res.end('<!doctype html><html lang="en"><meta charset="utf-8"><title>received</title>'
        + '<pre id="echo">POST ' + url.pathname + ' body: ' + body + '</pre>');
    });
    return;
  }
  res.writeHead(200, { 'content-type': 'text/html; charset=utf-8', 'cache-control': 'no-store' });
  if (url.pathname === '/payment') return res.end(cardPage(url.searchParams.get('variant') || 'spec'));
  res.end(accountPage());
}).listen(PORT, '127.0.0.1', () => console.log('form target on 127.0.0.1:' + PORT));

probe.mjs, the 54 field names, the grammar, and the browser driver:

// probe.mjs <case> : audit | fill | off | bogus | pseudo
// Reads every autocomplete attribute against the HTML standard's autofill field names,
// and drives one real browser fill through CDP so input:autofill has something to match.
import { launch } from 'puppeteer-core';

const CHROME = process.env.CHROME ?? 'C:/Program Files/Google/Chrome/Application/chrome.exe';
const BASE = 'http://127.0.0.1:8934';
const kase = process.argv[2] ?? 'audit';
const wait = (ms) => new Promise((r) => setTimeout(r, ms));

// The 54 autofill field names, by control group, from the HTML standard's table.
const GROUPS = {
  text: ('name honorific-prefix given-name additional-name family-name honorific-suffix nickname'
    + ' organization-title organization address-line1 address-line2 address-line3 address-level4'
    + ' address-level3 address-level2 address-level1 country country-name postal-code cc-name'
    + ' cc-given-name cc-additional-name cc-family-name cc-number cc-csc cc-type transaction-currency'
    + ' language sex tel-country-code tel-national tel-area-code tel-local tel-local-prefix'
    + ' tel-local-suffix tel-extension').split(' '),
  multiline: ['street-address'],
  password: ['new-password', 'current-password', 'one-time-code'],
  username: ['username', 'email'],
  tel: ['tel'],
  numeric: ['cc-exp-month', 'cc-exp-year', 'transaction-amount', 'bday-day', 'bday-month', 'bday-year'],
  month: ['cc-exp'],
  date: ['bday'],
  url: ['url', 'photo', 'impp'],
};
// Every group also accepts hidden, text, search, textarea and select; multiline does not take text or search.
const EXTRA = { text: [], multiline: [], password: ['password'], username: ['email'], tel: ['tel'],
  numeric: ['number'], month: ['month'], date: ['date'], url: ['url'] };
const CONTACT = ['tel', 'tel-country-code', 'tel-national', 'tel-area-code', 'tel-local',
  'tel-local-prefix', 'tel-local-suffix', 'tel-extension', 'email', 'impp'];
const MODES = ['home', 'work', 'mobile', 'fax', 'pager'];

const groupOf = (tok) => Object.keys(GROUPS).find((g) => GROUPS[g].includes(tok)) || null;
const allowed = (g) => (g === 'multiline'
  ? ['hidden', 'textarea', 'select-one', 'select-multiple']
  : ['hidden', 'text', 'search', 'textarea', 'select-one', 'select-multiple'].concat(EXTRA[g]));

function verdict(raw, type) {
  if (raw === null) return 'no attribute';
  const v = raw.trim().toLowerCase();
  if (v === '') return 'empty attribute';
  const t = v.split(' ').filter(Boolean);
  if (t.length === 1 && (t[0] === 'on' || t[0] === 'off')) return t[0];
  const used = [];
  let i = 0;
  if (t[i] && t[i].startsWith('section-') && t[i].length > 8) { used.push(t[i]); i += 1; }
  if (t[i] === 'shipping' || t[i] === 'billing') { used.push(t[i]); i += 1; }
  let mode = null;
  if (MODES.includes(t[i])) { mode = t[i]; used.push(mode); i += 1; }
  const field = t[i];
  i += 1;
  if (t[i] === 'webauthn') { used.push('webauthn'); i += 1; }
  if (i < t.length) return 'unparsed token: ' + t[i];
  const g = groupOf(field);
  if (!g) return 'invented token: ' + field;
  if (mode && !CONTACT.includes(field)) return 'mode ' + mode + ' needs tel/email';
  if (!allowed(g).includes(type)) return 'wrong control: ' + type + ' not in group ' + g;
  return used.length ? 'ok (' + used.join(' ') + ')' : 'ok';
}

function table(rows, cols) {
  const w = cols.map((c) => Math.max(c.length, ...rows.map((r) => String(r[c]).length)));
  const line = (cells) => cells.map((c, i) => String(c).padEnd(w[i])).join('  ').trimEnd();
  console.log(line(cols));
  for (const r of rows) console.log(line(cols.map((c) => r[c])));
}

const readControls = (page) => page.evaluate(() => [...document.querySelectorAll('#f [name]')].map((el) => ({
  name: el.name,
  id: el.id,
  type: el.type,
  label: (document.querySelector('label[for="' + el.id + '"]') || {}).textContent || '-',
  autocomplete: el.getAttribute('autocomplete'),
})));

const countFilled = (page) => page.evaluate(() => ({
  n: document.querySelectorAll('input:autofill').length,
  ids: [...document.querySelectorAll('input:autofill')].map((el) => el.id).join(',') || '-',
}));

const browser = await launch({ executablePath: CHROME, headless: true, args: ['--lang=en-US'] });
const page = await browser.newPage();
const cdp = await page.createCDPSession();
await cdp.send('Autofill.enable');
console.log('case ' + kase);

async function fillCard(variant, submit) {
  await page.goto(BASE + '/payment?variant=' + variant, { waitUntil: 'load' });
  await page.evaluate(() => {
    window.seen = {};
    for (const el of document.querySelectorAll('#pay input')) {
      for (const type of ['input', 'change', 'keydown']) {
        el.addEventListener(type, () => {
          window.seen[el.id + ':' + type] = (window.seen[el.id + ':' + type] || 0) + 1;
        });
      }
    }
  });
  const doc = await cdp.send('DOM.getDocument');
  const { nodeId } = await cdp.send('DOM.querySelector', { nodeId: doc.root.nodeId, selector: '#ccnum' });
  const { node } = await cdp.send('DOM.describeNode', { nodeId });
  const frameId = (await cdp.send('Page.getFrameTree')).frameTree.frame.id;
  await page.focus('#ccnum');
  await cdp.send('Autofill.trigger', {
    fieldId: node.backendNodeId,
    frameId,
    card: { number: '4444444444444444', name: 'Ada Lovelace', expiryMonth: '12', expiryYear: '2030', cvc: '123' },
  });
  await wait(1200);
  const rows = await page.evaluate(() => [...document.querySelectorAll('#pay input')].map((el) => ({
    id: el.id,
    autocomplete: el.getAttribute('autocomplete'),
    'el.value': el.value === '' ? '(empty)' : el.value,
    ':autofill': String(el.matches(':autofill')),
    events: ['input', 'change', 'keydown'].map((t) => t + '=' + (window.seen[el.id + ':' + t] || 0)).join(' '),
  })));
  table(rows, ['id', 'autocomplete', 'el.value', ':autofill', 'events']);
  if (submit) {
    await Promise.all([page.waitForNavigation({ waitUntil: 'load' }), page.click('#pay-go')]);
    console.log(await page.$eval('#echo', (el) => el.textContent));
  }
}

if (kase === 'audit') {
  await page.goto(BASE + '/', { waitUntil: 'load' });
  const rows = (await readControls(page)).map((r) => ({
    ...r,
    autocomplete: r.autocomplete === null ? '(none)' : r.autocomplete,
    verdict: verdict(r.autocomplete, r.type),
  }));
  table(rows, ['name', 'id', 'type', 'label', 'autocomplete', 'verdict']);
  const bad = rows.filter((r) => !r.verdict.startsWith('ok')).length;
  console.log(rows.length + ' controls, ' + bad + ' not ok, checked against 54 field names');
} else if (kase === 'fill') {
  await fillCard('spec', true);
} else if (kase === 'off') {
  await fillCard('off', true);
} else if (kase === 'bogus') {
  await fillCard('bogus', true);
} else if (kase === 'pseudo') {
  await page.goto(BASE + '/', { waitUntil: 'load' });
  console.log('on load: input:autofill = ' + (await countFilled(page)).n);
  await page.type('#fullname', 'Ada Lovelace');
  await page.evaluate(() => { document.querySelector('#city').value = 'Springfield'; });
  const a = await countFilled(page);
  console.log('after typing, and after el.value = "Springfield": input:autofill = ' + a.n + ', ids ' + a.ids);
  await fillCard('spec');
  const b = await countFilled(page);
  console.log('after the browser filled the card form: input:autofill = ' + b.n + ', ids ' + b.ids);
  await page.type('#ccnum', '0');
  const c = await countFilled(page);
  console.log('after typing one digit into ccnum: input:autofill = ' + c.n + ', ids ' + c.ids);
  await page.evaluate(() => { document.querySelector('#ccname').value = 'Someone Else'; });
  const d = await countFilled(page);
  console.log('after el.value on ccname:          input:autofill = ' + d.n + ', ids ' + d.ids);
}

console.log('chrome ' + (await browser.version()));
await browser.close();

Steps

  1. Step 1.

    Start the target in its own shell.

    node server.mjs
    
    form target on 127.0.0.1:8934

    Stop it afterwards by the PID that owns port 8934, not by image name.

  2. Step 2.

    Audit every control on the account form.

    node probe.mjs audit
    
    case audit
    name      id          type        label             autocomplete                         verdict
    fullname  fullname    text        Full name         name                                 ok
    email     email       email       Email             email                                ok
    phone     phone       tel         Phone             tel-number                           invented token: tel-number
    country   country     select-one  Country           country                              ok
    city      city        text        City              address-line1                        ok
    zip       zip         number      ZIP               shipping postal-code                 wrong control: number not in group text
    street    street      textarea    Street            shipping street-address              ok (shipping)
    billzip   billzip     text        Billing ZIP       section-billing billing postal-code  ok (section-billing billing)
    account   account     text        Username          username1                            invented token: username1
    q         sitesearch  search      Search                                                 empty attribute
    newpass   newpass     password    New password      new-password                         ok
    newpass2  confirm     password    Repeat password   new-password                         ok
    oldpass   oldpass     password    Current password  current-password                     ok
    otp       otp         text        SMS code          one-time-code                        ok
    cvc       cvc         text        Security code     off                                  off
    promo     promo       text        Promo code        (none)                               no attribute
    news      news        checkbox    Email me          email                                wrong control: checkbox not in group username
    worktel   worktel     tel         Work phone        work tel                             ok (work)
    lastname  lastname    text        Last name         work family-name                     mode work needs tel/email
    19 controls, 8 not ok, checked against 54 field names
    chrome Chrome/152.0.7977.76

    Eight rows are not ok: two invented tokens, two tokens on a control type their group forbids, one empty attribute, one absent, one off, and one contact mode before a field name that takes none. section-billing billing postal-code and work tel pass, so prefixes are checked too. The password pair and one-time-code pass as tokens, and no more than that.

  3. Step 3.

    Fill the card form with the tokens the standard defines.

    node probe.mjs fill
    
    case fill
    id      autocomplete  el.value          :autofill  events
    ccname  cc-name       Ada Lovelace      true       input=1 change=1 keydown=1
    ccnum   cc-number     4444444444444444  true       input=1 change=1 keydown=1
    ccexp   cc-exp        12/2030           true       input=1 change=1 keydown=1
    cvc     cc-csc        123               true       input=1 change=1 keydown=1
    POST /pay body: ccname=Ada+Lovelace&ccnum=4444444444444444&ccexp=12%2F2030&cvc=123
    chrome Chrome/152.0.7977.76

    Four values, :autofill true on each, and one input, one change and one keydown on every field, counted by listeners the program attaches before the fill. The last line is what the server received.

  4. Step 4.

    Serve the same four fields with autocomplete="off" on each.

    node probe.mjs off
    
    case off
    id      autocomplete  el.value          :autofill  events
    ccname  off           Ada Lovelace      true       input=1 change=1 keydown=1
    ccnum   off           4444444444444444  true       input=1 change=1 keydown=1
    ccexp   off           12/2030           true       input=1 change=1 keydown=1
    cvc     off           123               true       input=1 change=1 keydown=1
    POST /pay body: ccname=Ada+Lovelace&ccnum=4444444444444444&ccexp=12%2F2030&cvc=123
    chrome Chrome/152.0.7977.76

    Same four values, :autofill true on all four, and a POST body matching step 3 character for character. off changed nothing this run can see.

  5. Step 5.

    Serve the same fields with tokens that are not in the list.

    node probe.mjs bogus
    
    case bogus
    id      autocomplete  el.value          :autofill  events
    ccname  cardholder    Ada Lovelace      true       input=1 change=1 keydown=1
    ccnum   card-number   4444444444444444  true       input=1 change=1 keydown=1
    ccexp   cc-expiry     12/2030           true       input=1 change=1 keydown=1
    cvc     cvc2          123               true       input=1 change=1 keydown=1
    POST /pay body: ccname=Ada+Lovelace&ccnum=4444444444444444&ccexp=12%2F2030&cvc=123
    chrome Chrome/152.0.7977.76

    cardholder, card-number, cc-expiry and cvc2 read like field names and are none. Chrome filled all four from the labels and name attributes, so watching a fill says nothing about the tokens.

  6. Step 6.

    Read the pseudo-class for what the browser filled.

    node probe.mjs pseudo
    
    case pseudo
    on load: input:autofill = 0
    after typing, and after el.value = "Springfield": input:autofill = 0, ids -
    …
    after the browser filled the card form: input:autofill = 4, ids ccname,ccnum,ccexp,cvc
    after typing one digit into ccnum: input:autofill = 3, ids ccname,ccexp,cvc
    after el.value on ccname:          input:autofill = 2, ids ccexp,cvc
    chrome Chrome/152.0.7977.76

    Zero at load, zero after page.type and after an assignment to el.value, the two ways a test framework fills a form. Four after the browser filled, three after one typed digit, two after a script overwrote another field. The pseudo-class gives a field up the moment anything else touches it.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | invented token: tel-number | The value is not one of the 54 field names, so the attribute promises nothing | Replace it with the token the standard defines, tel here. | | wrong control: number not in group text | The token is real and its control group does not include this type | Change the type, or move to a token whose group allows it. | | empty attribute or no attribute | Nothing is declared, and the browser is left with its heuristics | Decide the field's token, or record that the field is deliberately unlabelled. | | mode work needs tel/email | home, work, mobile, fax and pager are defined only in front of a tel*, email or impp field name | Drop the mode, or move it to the contact field it belongs to. | | ok on a row whose label disagrees with the token | The grammar and the control type passed. Meaning is not checkable by a script | Read every ok row against its label. city carrying address-line1 is in step 2. | | input:autofill returns 0 in a test | The values were typed or assigned, not filled by the browser | Assert on the values, and keep autofill assertions for a session that really fills. | | Two POST bodies identical across variants | The attribute is not on the wire, and no server rule changes with it | Keep server rules in How to test that client validation is enforced on the server. |

Common mistakes

Sign: The audit comes back green, so the form's autofill is signed off.Cause: A script can check grammar and control type, not meaning. The row named city, labelled City, carrying address-line1, passed as ok in step 2, and address-line1 is a street address. Read the ok rows against the labels beside them.
Sign: A field holds data it should not, and the fix in the ticket is autocomplete=off.Cause: Step 4 put off on all four card fields. Chrome filled all four with the same values as step 3, matched them all with input:autofill, and the server received an identical body. Whatever off is for, it did not stop this fill.
Sign: Autofill was tested by hand on a profile that has an address saved, and everything filled.Cause: Step 5 filled four fields whose tokens were cardholder, card-number, cc-expiry and cvc2, none of which exist. The browser fills from labels and names as well as tokens, so a successful fill is not evidence that any token is correct.
Sign: An end-to-end test asserts input:autofill after filling the form, and the assertion is always 0.Cause: Step 6 measured 0 after page.type and after el.value, and 4 only after the browser itself filled. The count also fell to 3 when one digit was typed into a filled field. The pseudo-class answers what the browser did, not what your test did.

What to check next

FAQ

How do you check autofill form data?

Read the attributes first, with step 2. Then fill the form and read input:autofill, which named ccname,ccnum,ccexp,cvc in step 6 and returned 0 for values the harness typed or assigned.

How do you test browser autofill without a saved profile?

Test the half that does not need one. The token audit runs anywhere, and Autofill.trigger fills a card form with data you supply. The suggestion list and the password manager need a profile.

Does autocomplete="off" stop the browser filling a field?

Not in step 4. Four fields carrying off took the same values as the run with correct tokens, and the two request bodies matched.

What does the attribute change about the request?

Nothing. Steps 3, 4 and 5 sent three different sets of attributes and produced one identical POST body, ccname=Ada+Lovelace&ccnum=4444444444444444&ccexp=12%2F2030&cvc=123. Rules about the value belong on the server.

Which autofill behaviour could this run not measure?

Whether Chrome offers a generated password on new-password or saves the pair, whether one-time-code takes a code from a message, and what a suggestion shows before it is chosen. Each needs a credential store, a device or a person.

Verified

Verified by Maks Vernynode 22.23.2Chrome 152.0.7977.76puppeteer-core 25.10.0

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

intermediate12 minpublished updated Maks Verny