How to view a page as googlebot

Send the request with Googlebot's user agent and compare the bytes against a normal fetch: curl -sS -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' URL. That shows what the server chooses to return to that string. It is not Google's view, and it says nothing about what Google rendered.

Why check this

Run this on staging sign-off and after any change to a CDN rule, an edge worker, a bot-management product or a prerender service, because all four can branch on the request header. The failure it prevents is a branch nobody meant to ship: a prerender layer that answers crawler-shaped requests with a stale snapshot, or a WAF rule that hands bot traffic a challenge page while every human tester sees a working site.

The check settles one question: given a request carrying that string, what bytes come back, and do they differ from the bytes a browser string gets. The string is a header the client writes, so sending it makes you a client claiming to be Googlebot. Only Search Console URL Inspection reports what Google fetched and rendered, on a verified property.

Prerequisites

• curl with the -A flag. See the curl manual.
• The two user agent strings written out, since a truncated one branches differently from a complete one. Google publishes the current list in Google crawlers.
• The meta tag checker if you want the served HTML parsed rather than read by eye. It reads the response and never runs page scripts.
• A local server for the branching case. This one answers the same URL two ways, and is the only host in this procedure that treats the header as a switch:

import { createServer } from 'node:http';
const BOT = /googlebot|bingbot|yandex/i;
const plans = 'Team 49, Business 99, Enterprise on request';
const body = (mode) => `<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Pricing</title></head>
<body><h1>Pricing</h1><p id="mode">served-to: ${mode}</p>
${mode === 'crawler'
  ? `<p id="plans">${plans}</p>`
  : `<p id="plans"></p><script>document.getElementById('plans').textContent='${plans}';</script>`}
</body></html>`;
createServer((req, res) => {
  const mode = BOT.test(req.headers['user-agent'] ?? '') ? 'crawler' : 'browser';
  res.writeHead(200, { 'content-type': 'text/html; charset=utf-8', vary: 'User-Agent' });
  res.end(body(mode));
}).listen(8817, '127.0.0.1', () => console.log('ua-server listening on 127.0.0.1:8817'));

• A script that prints what Chrome ended up with, for step 6. The same file is used by the two procedures linked at the end:

import { launch } from 'puppeteer-core';
const args = process.argv.slice(2);
const opt = (n, d = '') => { const a = args.find((x) => x.startsWith(`--${n}=`)); return a ? a.slice(n.length + 3) : d; };
const browser = await launch({ executablePath: 'C:/Program Files/Google/Chrome/Application/chrome.exe', headless: true });
const page = await browser.newPage();
if (opt('ua')) await page.setUserAgent(opt('ua'));
await page.setJavaScriptEnabled(!args.includes('--nojs'));
await page.goto(args[0], { waitUntil: 'networkidle2', timeout: 30000 });
if (opt('click')) await page.click(opt('click'));
if (opt('wait')) await new Promise((r) => setTimeout(r, Number(opt('wait'))));
const sel = opt('count');
const out = sel
  ? `${await page.evaluate((s) => document.querySelectorAll(s).length, sel)}`
  : await page.evaluate((t) => (t ? document.body.innerText : document.documentElement.outerHTML), args.includes('--text'));
process.stdout.write(out.endsWith('\n') ? out : `${out}\n`);
await browser.close();

Browser figures below are one capture on one machine, Chrome 152.0.7977.76.

Steps

1. 1

   Step 1.

   Fetch a page you do not own with an ordinary desktop browser string, into a file.

   curl -sS -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36' https://developer.mozilla.org/en-US/docs/Glossary/SEO -o mdn-browser.html -w 'browser  %{http_code} %{size_download} bytes\n'
   

   browser  200 185364 bytes

2. 2

   Step 2.

   Repeat the request with Googlebot's string, into a second file.

   curl -sS -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' https://developer.mozilla.org/en-US/docs/Glossary/SEO -o mdn-bot.html -w 'googlebot %{http_code} %{size_download} bytes\n'
   

   googlebot 200 185364 bytes

   Equal lengths are suggestive, not conclusive. A branch that swaps one word for another of the same length keeps the byte count.

3. 3

   Step 3.

   Compare the two files by digest rather than by eye.

   sha256sum mdn-browser.html mdn-bot.html
   

   b69cdf4309d9cbf50b12cafa570aa56ee16d07a8bed8f70d19ce2c0b5502ee2e *mdn-browser.html
   b69cdf4309d9cbf50b12cafa570aa56ee16d07a8bed8f70d19ce2c0b5502ee2e *mdn-bot.html

   One digest for both strings. On 2026-09-11 that host served a single variant, which is the result to expect on a site that does not branch.

4. 4

   Step 4.

   Run the same pair against the local server, which does branch, and diff the bodies.

   curl -sS http://127.0.0.1:8817/ -o default.html \
     && curl -sS -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' http://127.0.0.1:8817/ -o googlebot.html \
     && diff default.html googlebot.html
   

   3,4c3,4
   < <body><h1>Pricing</h1><p id="mode">served-to: browser</p>
   < <p id="plans"></p><script>document.getElementById('plans').textContent='Team 49, Business 99, Enterprise on request';</script>
   ---
   > <body><h1>Pricing</h1><p id="mode">served-to: crawler</p>
   > <p id="plans">Team 49, Business 99, Enterprise on request</p>

   Two differences, and only one matters. The served-to marker is a label this server prints. The real change is the second line: the crawler string gets the prices as markup, the browser string gets an empty element plus a script.

5. 5

   Step 5.

   Read the response headers on the branching host and look for the declaration.

   curl -sS -D - -o /dev/null http://127.0.0.1:8817/
   

   HTTP/1.1 200 OK
   content-type: text/html; charset=utf-8
   vary: User-Agent
   Date: Fri, 11 Sep 2026 20:42:40 GMT

   vary: User-Agent is the server admitting the response depends on the header. Strong as a positive signal, weak as a negative one: a server can branch without sending it.

6. 6

   Step 6.

   Load the same URL in Chrome under both strings and compare the visible text after scripts run.

   node render.mjs http://127.0.0.1:8817/ --text > dom-browser.txt \
     && node render.mjs http://127.0.0.1:8817/ --text --ua='Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' > dom-bot.txt \
     && diff dom-browser.txt dom-bot.txt
   

   3c3
   < served-to: browser
   ---
   > served-to: crawler

   The prices line is gone from the diff. Both loads end with the same text, because the browser variant builds it with a script. Only the marker line survives, and a real site would not print one.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | Identical digests across both strings | The server returns one variant to that string | Move on to the rendering comparison | | Bodies differ and vary: User-Agent is present | A deliberate branch, declared to caches | Find the rule and confirm someone meant it | | Bodies differ and vary is absent | An undeclared branch | Fix the header first: a shared cache will serve one variant to everyone | | A 403, a challenge page or a 429 under the bot string only | A bot-management layer scored the request | This is a finding about that layer, not about crawlability | | Both strings return the same empty shell | Nothing is branching, and nothing is server-rendered either | Go to the server-side rendering check |

Common mistakes

What to check next

• How to check if googlebot can render javascript: the same two captures, used to count what only exists after scripts run.
• How to check if server side rendering is working: the inverse question, whether the served bytes already carry the content.
• How to check robots.txt: a crawler reads that file before it reads this page, so check it first when a fetch behaves oddly.
• How to check if a page is indexable: the directives that decide what happens after the fetch succeeds.

FAQ

How to check website crawling on a whole site rather than one page?

Run the pair of requests across a URL list and compare digests per URL, reporting only the URLs where they differ. Keep the rate to something your own origin is happy to serve.

How to detect search engine bots on the server side?

Not by the user agent, which step 3 shows is a client-supplied switch. Google documents verifying its crawlers by reverse DNS on the requesting IP address, then a forward lookup of the returned host name back to the same address. The header is a hint, the address is the evidence.

Does this show me what Google rendered?

No. It shows what your server returned to a request carrying that string. Google fetches on its own schedule, from its own addresses, with its own render timeouts. URL Inspection in Search Console reports the fetched HTML and the rendered DOM, on a verified property.

Is serving different HTML to Googlebot a problem?

It is a decision that needs a reason on record. Google treats content shown to crawlers but not to users as cloaking. Device and locale branches are ordinary, so a branch is not itself the finding. A branch nobody can explain is.

Verified