h2check

How to check nofollow links

Take an inventory rather than spot-checking one anchor. Run node link-rel.cjs https://host/page, which prints every link on the page with its rel value and whether it points inside the site. Paid and user-generated links should carry a marker, and internal links should not carry one by accident.

Why check this

Run this on any template that renders links from data: a blog post, a comments thread, a partner directory, a sponsored placement. Run it again after a CMS upgrade, because the attribute is usually added by a plugin and lost by a theme.

Two failures sit at opposite ends. A paid placement ships without a marker, which is the case the marker exists for. Or a navigation partial adds rel="nofollow" to internal links, and the site asks search engines not to follow its own routes. The second is harder to notice, because nothing on the page looks different.

The meaning of the attribute changed, and the dates matter. rel="nofollow" was introduced in 2005 as an instruction. On 2019-09-10 Google introduced rel="sponsored" for paid links and rel="ugc" for user-generated ones, and made all three work as hints for ranking. From 2020-03-01 nofollow became a hint for crawling and indexing as well. A hint can be overruled, so a nofollowed URL may still be crawled and may still be indexed.

Prerequisites

Save this as links-server.cjs and start it with node links-server.cjs.

// node links-server.cjs   ->   http://127.0.0.1:8839
// /blog/ carries a mix of marked and unmarked outbound links.
// /comments/ carries a page-level nofollow that overrides every anchor on it.
const http = require('http');
const PORT = 8839;

const links = `
<a href="https://ads.example.com/deal" rel="sponsored">paid placement</a>
<a href="https://forum.example.com/post/1" rel="ugc">reader comment</a>
<a href="https://partner.example.com/" rel="nofollow">partner</a>
<a href="https://news.example.org/story">press coverage</a>
<a href="/pricing/" rel="nofollow">our pricing</a>
<a href="/docs/">our docs</a>
<a href="/login" rel="nofollow noopener">sign in</a>`;

const page = (head) =>
  `<!doctype html>\n<html lang="en">\n<head>\n<meta charset="utf-8">\n<title>Blog</title>\n${head}\n</head>\n<body>\n${links}\n</body>\n</html>\n`;

http.createServer((req, res) => {
  const head = req.url.startsWith('/comments/') ? '<meta name="robots" content="nofollow">' : '';
  res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
  res.end(page(head));
}).listen(PORT, '127.0.0.1', () => console.log(`links server on ${PORT}`));

Save this as link-rel.cjs.

// node link-rel.cjs <url> [--summary]
// Inventory of every anchor in the served HTML with its rel value and its scope.
const url = new URL(process.argv[2]);
const summary = process.argv.includes('--summary');

(async () => {
  const html = await (await fetch(url)).text();
  const meta = /<meta[^>]+name=["']?robots["']?[^>]*>/i.exec(html);
  const page = meta && (/content=["']([^"']*)["']/i.exec(meta[0]) || [])[1];
  console.log(`${url.href}\npage-level robots: ${page || '(none)'}`);

  const rows = [];
  for (const a of html.matchAll(/<a\s([^>]*)>/gi)) {
    const href = (/href=["']([^"']*)["']/i.exec(a[1]) || [])[1];
    if (!href || href.startsWith('#') || /^(mailto|tel|javascript):/i.test(href)) continue;
    const rel = ((/rel=["']([^"']*)["']/i.exec(a[1]) || [])[1] || '').toLowerCase().trim();
    const scope = new URL(href, url).host === url.host ? 'internal' : 'external';
    rows.push({ rel: rel || '(none)', scope, href });
  }

  if (summary) {
    const tally = {};
    for (const r of rows) for (const t of r.rel.split(/[\s,]+/)) tally[`${r.scope} ${t}`] = (tally[`${r.scope} ${t}`] || 0) + 1;
    for (const k of Object.keys(tally).sort()) console.log(`  ${String(tally[k]).padStart(4)}  ${k}`);
  } else {
    for (const r of rows) console.log(`  ${r.rel.padEnd(17)} ${r.scope.padEnd(9)} ${r.href}`);
  }
  const ext = rows.filter((r) => r.scope === 'external');
  console.log(`${rows.length} links: ${ext.length} external, ${rows.length - ext.length} internal, ` +
    `${rows.filter((r) => /nofollow|sponsored|ugc/.test(r.rel)).length} carry nofollow, sponsored or ugc`);
})();

Steps

  1. Step 1.

    Tally the rel attributes on the page with one command, to see the shape of the problem.

    curl -sS http://127.0.0.1:8839/blog/ | grep -o 'rel="[^"]*"' | sort | uniq -c

          1 rel="nofollow noopener"
    2 rel="nofollow"
    1 rel="sponsored"
    1 rel="ugc"

    Five attributes on a page with seven links. The two links carrying no rel at all are invisible to this command, and they are the ones an audit is looking for.

  2. Step 2.

    Run the full inventory, which lists the unmarked links as well.

    node link-rel.cjs http://127.0.0.1:8839/blog/

    http://127.0.0.1:8839/blog/
page-level robots: (none)
sponsored         external  https://ads.example.com/deal
ugc               external  https://forum.example.com/post/1
nofollow          external  https://partner.example.com/
(none)            external  https://news.example.org/story
nofollow          internal  /pricing/
(none)            internal  /docs/
nofollow noopener internal  /login
7 links: 4 external, 3 internal, 5 carry nofollow, sponsored or ugc

    Read the internal rows first. /pricing/ is a page of this site asking not to be followed, which is the own-goal: an editor copied an anchor from an external link and kept its attribute. /login is the deliberate case, since a sign-in route has nothing to index.

  3. Step 3.

    Run the same inventory on a page that carries a page-level directive.

    node link-rel.cjs http://127.0.0.1:8839/comments/

    http://127.0.0.1:8839/comments/
page-level robots: nofollow
sponsored         external  https://ads.example.com/deal
ugc               external  https://forum.example.com/post/1
nofollow          external  https://partner.example.com/
(none)            external  https://news.example.org/story
nofollow          internal  /pricing/
(none)            internal  /docs/
nofollow noopener internal  /login
7 links: 4 external, 3 internal, 5 carry nofollow, sponsored or ugc

    Identical anchors, one different line. page-level robots: nofollow applies to every link in the document, so the per-link values no longer decide anything. A tool that reads only anchors reports this page as partly followed.

  4. Step 4.

    Run the summary form against a real page, to see what a site with no paid links looks like.

    node link-rel.cjs https://developer.mozilla.org/en-US/docs/Web/HTTP --summary

    https://developer.mozilla.org/en-US/docs/Web/HTTP
page-level robots: (none)
  18  external (none)
  10  external noopener
 538  internal (none)
   2  internal noopener
568 links: 28 external, 540 internal, 0 carry nofollow, sponsored or ugc

    Not one of 568 links is nofollowed, and that is the correct result for a documentation site with no advertising and no user submissions. The 12 noopener values are a window security attribute sharing the same rel namespace, and they say nothing about crawling.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | (none) on an external link that is paid | An advertisement is presented as an editorial link | Add rel="sponsored" in the template that renders the placement | | (none) on an external link in user content | Comments and profiles are unmarked | Add rel="ugc" in the renderer, not per comment | | nofollow on an internal row | The site is hinting against its own pages | Remove it unless the route is a login, a cart or a filtered view | | noopener or noreferrer only | A window and referrer control, not a crawl hint | Leave it. It has no effect on link qualification | | page-level robots: nofollow | Every link in the document is hinted, anchors included | Decide at the page level, then stop editing anchors |

Thresholds

rel=nofollow has been a hint rather than a directive for ranking since 2019-09-10, and for crawling and indexing since 2020-03-01. rel=sponsored and rel=ugc were introduced on the same 2019-09-10 announcement. Source: https://developers.google.com/search/blog/2019/09/evolving-nofollow-new-ways-to-identify

Common mistakes

Sign: An audit tool reports every link on the page as followed, and the anchors clearly carry rel values.Cause: The tool read the rendered DOM after a consent script replaced the link markup, or it read the served HTML while the links are injected by a script. Compare both. Step 2 reads what the server sent, which is what a crawler that does not execute scripts receives.
Sign: Paid links are marked, and the marking is undone on one template.Cause: A page-level robots meta with a follow value, or an X-Robots-Tag header, sits above the anchors. Step 3 shows the page-level line for that reason. Per-link values and page-level values are two separate channels, and a checklist that reads only one of them passes a page it should fail.
Sign: Nofollow was added to keep a page out of search results, and the page is in the index.Cause: nofollow applies to a link, not to the page it points at. The target can be reached from a sitemap, a redirect or another site, and since 2020-03-01 the value is a hint even for crawling. Keeping a page out of an index is a noindex, on that page.

What to check next

FAQ

How to check dofollow and nofollow links?

There is no dofollow value. A link without nofollow, sponsored or ugc is followed by default, so the inventory in step 2 answers both questions at once: the (none) rows are the followed links, and the rest are hinted.

When to use rel external or rel nofollow?

external says the target is outside the current site and is not part of Google's link qualification at all. Use sponsored for paid placements, ugc for user-submitted content, and nofollow when neither fits but you want no association with the target.

Does nofollow stop the linked page from being indexed?

No. It is a hint about one link, and since 2020-03-01 it is a hint for crawling and indexing rather than an instruction. The target can be discovered from a sitemap or from any other site. Use a noindex on the target page instead.

Should internal links ever carry nofollow?

Rarely. Login, logout, cart and faceted filter URLs are the usual cases, and a noindex on those pages is the stronger control. A nofollow on a normal internal route, as in step 2, is almost always a copy and paste accident.

Do sponsored and ugc replace nofollow?

They sit alongside it. All three can appear together in one space-separated rel value, and a link that already carries nofollow keeps working. Add the more specific value when the link is paid or user-submitted, since it describes the relationship rather than the intention.

Verified

Verified by Maks Vernycurl 8.21.0node 22.23.2

Each output block is what the command above it printed on that date, on the host named in the step. Figures read from a live site move between runs. Compare the shape of the answer rather than the digits, and see the methodology for how a page is re-verified.

basic6 minpublished updated Maks Verny