Free browser tool / Googlebot text search
Robots meta tag reference: every directive
Every robots meta directive that matters in 2026, in one table: syntax, effect, and the default when the tag is absent. Directives are case-insensitive and comma-separated. Unknown values are ignored, so a typo silently disables the whole intent.
The full directive table
| Directive | Effect | Default if absent |
|---|---|---|
| index | Page may be indexed | Yes, indexed |
| noindex | Drop from index, do not store | Indexed |
| follow | Crawl links on the page | Yes, followed |
| nofollow | Do not crawl links on the page | Followed |
| none | Same as noindex, nofollow | Both allowed |
| noarchive | No cached copy in search results | Cache allowed |
| nosnippet | No text snippet or video preview | Snippet allowed |
| max-snippet: N | Snippet capped at N characters, 0 means none | Unspecified |
| max-image-preview: setting | none, standard, large, or none-larger for AMP | Unspecified |
| max-video-preview: N | Video preview capped at N seconds | Unspecified |
| notranslate | No translation offered in results | Translation allowed |
| noimageindex | Page images not indexed | Images indexed |
| unavailable_after: RFC date | Drop from results after the date | No expiry |
| nositelinkssearchbox | No sitelinks search box | Box possible |
| indexifembedded | Index even with noindex when embedded via iframe | Noindex wins |
Where the tag goes
The tag sits in the head of the document. Google reads the first head block it finds, even if the document has a second head later in the markup, and it does not read robots meta in the body. One tag per page is the safe pattern; if two tags on the same page conflict, Google applies the most restrictive combination across them.
If you do not specify a tag at all, the effective value is index, follow. Explicitly writing that is legal but redundant.
User-agent overrides
Each user-agent line is an independent grant. You can tighten rules per bot:
<meta name="robots" content="nofollow"> <meta name="googlebot" content="noindex"> <meta name="googlebot-news" content="nosnippet">
Google applies the union of the most restrictive directives it reads for itself. A directive under googlebot applies to Googlebot only; every other engine falls back to the robots line. Bing reads bingbot the same way. Directives Bing does not support, like unavailable_after, are ignored by that engine even when Google honors them.
Copy-paste starter head block
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Page title here</title> <meta name="description" content="One or two sentences that a result page can show as the snippet."> <meta name="robots" content="index, follow"> </head>
For a page that must stay out of Google but stay crawlable for other engines' link discovery:
<meta name="googlebot" content="noindex">
What the tag does not do
- It does not stop crawling
- The page is fetched first, then parsed. Save crawl budget with robots.txt, not noindex.
- It does not work on PDFs or images
- Use the X-Robots-Tag header for anything that is not HTML.
- It does not act instantly
- Directives apply at recrawl time. Until then the old state persists in the index.
- It does not override canonical
- Conflicting noindex plus canonical gives Google two contradictory instructions; results are unpredictable. Choose one.
Verify any page in one paste: the meta robots checker parses the head of a live URL and shows which directives Google will read, with a companion FAQ for edge cases.