Free browser tool / Googlebot text search
Meta robots FAQ: 12 answers that hold up
The questions that actually come up when a page will not deindex, plus the answers that are true regardless of which blog you read. Every answer follows the same rule: fetch first, parse second, index third.
- Why is my page still in Google after I added noindex?
- Directives apply at recrawl time. Google has to fetch the page again before it reads the new tag. Days to weeks depending on how often the page changes; request a recrawl in Search Console to shorten it. If the page is blocked in robots.txt, Google never re-fetches it and the noindex never runs, which is the most common cause of a stuck listing.
- Does robots.txt prevent indexing?
- No. It prevents crawling. Google keeps URLs it learned from links and can list them without a snippet. To leave the index, a URL must be crawlable and carry noindex.
- Where does the meta robots tag go?
- Inside the head element, before the body starts. Google reads the first head block anywhere in the document, but tags in the body are ignored by validators and some parsers, so keep one head, one tag.
- Can I combine noindex and nofollow?
- Yes:
content="noindex, nofollow". The page leaves the index and its links are not crawled for discovery. Link-level nofollow on individual anchors is separate and does not need the meta tag. - Can I use noindex and canonical together?
- You can, but you should not. Canonical points Google at a preferred copy; noindex tells it to store nothing. Google resolves the conflict unpredictably and usually keeps one signal and discards the other. Pick the intent: keep the page and canonicalize it, or drop it with noindex.
- Does Google always honor noindex?
- When it can fetch and parse the page, yes. The exceptions are all access problems: robots.txt blocks, serve-time errors, geo or bot walls, or the tag appearing after the fetch that mattered. Google can also honor it late if the page rarely changes.
- How long until noindex takes effect?
- Until the next successful crawl of that page, then the next index refresh. Typically days to weeks. Popular pages recrawl faster. The URL Inspection tool shows the last crawl date and lets you request a live test.
- Does a meta robots tag in the body work?
- Google reads the first head-like block it encounters, even mid-document, so a stray tag sometimes works. Validators ignore it and the behavior is not contractual. Move it into the head.
- Should paginated pages be noindex?
- Usually not. Pagination in the index is normal and lets searchers land on deep items. noindex on page 2 onward also removes those URLs as crawl targets over time. Prefer canonical self-references and clean internal linking.
- How do I check a page's robots meta without viewing source?
- Open the page, or use the meta robots checker: paste a URL and it reads the live head, lists the directives, and flags conflicts like noindex plus canonical.
- What is the difference between none and noindex?
noneequals noindex plus nofollow. If you write both none and noindex, Google applies the union, which is the same thing. Prefer the explicit pair for readability.- Does noindex stop AI crawlers?
- No. The robots meta standard is about search indexing. AI crawlers like GPTBot or ClaudeBot are controlled in robots.txt by user-agent, and honoring it is each crawler's own policy, not a protocol guarantee.
Tool that answers the first question on any page: meta robots checker. Companion references: every directive in one table, the header equivalent for PDFs and site-wide rules, noindex versus robots.txt.