Read straight from the open capture script
What signals decide whether a page gets indexed?
Seven declared signals carry most of the weight. They are the HTTP status, the redirect chain, the canonical link, the robots meta tag, the X-Robots-Tag header, the Link response header, and whether the URL appears in the sitemap at all. Each one is a declaration a page makes about itself. None of them is a decision, because the decision belongs to the search engine.
This page reads each signal through the pipeline's own capture script, which is published unmodified at data/capture-indexing-signals-2026-09-08.py. Every limit quoted below is a constant in that file, not a rule of thumb.
- 7declared signals the script records for every URL
- 20URLs at most, the batch cap written into the script
- 2 MBresponse read cap, 2,000,000 bytes per fetch
- 30 stimeout on both the sitemap fetch and each page fetch
01The short answer, before any detail
A page gets indexed when a crawler can fetch it, understands that it should index it, and sees a reason to keep it. Seven declared signals sit on that path, and the capture script records all seven for every URL it visits.
- SIGNALHTTP status. A 200 says the page exists. A 404 or 410 says it does not, and a 5xx says try again later.
- SIGNALRedirect chain. The final URL after every hop is the URL the crawler actually sees.
- SIGNALCanonical link. The URL the page declares as its preferred version of itself.
- SIGNALRobots meta tag. A
robotsorgooglebotmeta element that can carrynoindex. - SIGNALX-Robots-Tag header. The same instruction delivered by response header instead of by markup.
- SIGNALLink response header. Relationship hints a server sends without touching the HTML.
- SIGNALSitemap membership. The script only ever visits URLs listed in a sitemap, so the sitemap is the first filter.
The script is explicit about what it is. Its own report field calls the population the first 20 URLs at most in the supplied sitemap and states, in the code, that this is not a representative sample of the web.
02How the script reads each signal, limits included
The script is 34 lines of Python with no dependency beyond the standard library. It fetches the sitemap, pulls out the loc text of every element ending in }loc, and walks the list. Here is what it does with each signal and where the code stops.
HTTP status and the final URL
Each URL is requested once with a named user agent, AIWebsitePipeline-SprintVerification/1.0, and urlopen follows redirects by default. The script then records r.status and r.url separately, so the status of the final response and the URL that served it are both kept. A chain of redirects collapses into one final URL in the record, and the intermediate hops are not retained.
Canonical link
A subclass of html.parser.HTMLParser scans the document. It appends to canonicals when it meets a link element whose rel contains the token canonical, and it stores both the href and an inHead flag showing whether the element appeared inside head. A canonical buried in the body is captured, and the flag lets you spot it.
Robots meta tag
The same parser collects any meta element whose name is robots or googlebot, in lower case, along with its content and the same inHead flag. Nothing else is read. A noindex delivered through any other element is invisible to this script.
X-Robots-Tag and Link headers
Response headers are recorded through headers.get_all() for both X-Robots-Tag and Link, so repeated headers survive into the JSON as a list rather than being flattened to one value. Nothing else in the response exposes a header-delivered instruction.
Sitemap membership
The sitemap itself is fetched with the same 30 second timeout and read through the same 2,000,000 byte cap. Only loc values beginning with http:// or https:// are kept. A relative or malformed loc is dropped before any fetch happens.
the real limits, as writtencapture-indexing-signals-2026-09-08.py
read cap r.read(2000000) # 2,000,000 bytes, page and sitemap timeout urlopen(..., timeout=30) # 30 seconds, sitemap and page batch cap for url in urls[:20] # first 20 URLs at most user agent AIWebsitePipeline-SprintVerification/1.0 no JS "No scripts executed" no sample "not a representative sample of the web" no state "Declarations are not Google-selected canonicals"
The script also hashes the raw bytes it reads with SHA-256 and keeps the digest rather than the body. The method field says the bytes are hashed and not retained, so the capture is a fingerprint of what a server returned at one moment, not an archive of it.
One more limit matters if you plan to repeat the run. The script takes a --sitemap argument that accepts a URL or a local file, and its own help text suggests using a saved pre-sprint sitemap to repeat the original population. The same inputs give the same population.
03Signal, what it controls, and how it fails
Read this table as a checklist for one URL. The middle column is what the signal can control. The right column is the failure that leaves the signal saying nothing useful.
| Signal | What it controls | Common failure |
|---|---|---|
| HTTP status | Whether the URL is treated as a real page, a missing page or a temporary error. | A 200 that serves an error page, or a soft 404 that says missing in the body and fine in the status line. |
| Redirect chain | Which URL ends up as the one a crawler can index. | Loops, or a chain long enough that the crawler gives up before reaching the target. |
| Canonical link | The preferred URL for a set of near duplicates, as declared by the page. | Several canonicals on one page, or a canonical pointing at a URL that itself redirects. |
| Robots meta tag | Whether the page asks crawlers to index it and follow its links. | A stray noindex left in a template that ships to every page. |
| X-Robots-Tag header | The same instruction, set at the server for a whole class of responses. | A header rule wide enough to catch HTML pages that were meant to be indexed. |
| Link header | Relationship hints such as a canonical or an alternate sent without markup. | A hint that contradicts the rel=canonical in the HTML, leaving the crawler to choose. |
| Sitemap membership | Which URLs a crawler is told to look at, and when they last changed. | URLs already gone, or lastmod values that change on every build and teach the crawler to ignore them. |
The last row is the one the script exposes most directly. Its population is the sitemap minus anything past the twentieth entry, so a page missing from the sitemap was never captured at all. Absence from the file is the quietest failure of the seven.
04Capturing these signals on your own pages
The script runs on your machine with python3 capture-indexing-signals-2026-09-08.py --sitemap <url or file> and writes indexing-signals.json. Point it at your own sitemap and you get the same seven signals back for your own URLs, with no account and nothing uploaded anywhere.
For a single page, three checkers on this site answer the same questions one signal at a time, in your browser.
- TOOLThe canonical tag checker resolves the URL a page declares as canonical, which is the third signal above.
- TOOLThe robots txt checker tests whether a crawler may fetch the path at all, which decides whether the robots meta tag is ever read.
- TOOLThe hreflang checker covers the alternate language hints, the signal family the Link header carries.
A capture is a snapshot. The script stamps every row with the time it was taken, so two runs tell you whether a signal changed rather than only what it is today.
05Questions this page gets asked
Does a canonical tag force Google to index the URL I picked?
No. A canonical tag is a hint. It tells Google which URL you would prefer, and Google weighs it against redirects, internal links, sitemap entries and duplicate content. The capture script records the canonical a page declares and never claims it is the URL Google selected.
Can a noindex meta tag remove a page that robots.txt already blocks?
No. A blocked URL is never crawled, so Google never reads the noindex on it. The page stays out of the index only because it was never fetched and other signals are absent. Remove the robots.txt block first, let the crawler fetch the page, and let the noindex do its work.
Why does the capture script stop after 20 URLs and 2 MB?
Those are the limits written into the script. It reads at most 20 URLs from the sitemap and reads at most 2,000,000 bytes of any single response, so the run stays bounded and finishes in a known time. The report states plainly that this is not a representative sample of the web.
Do these signals guarantee a page will be indexed?
No. Declared signals remove obstacles and give a crawler a clear instruction. They do not decide the outcome. A page can declare a clean canonical, return HTTP 200 and still sit out of the index if nothing links to it and no one searches for it. No traffic or indexation result is promised here.
06Where these claims come from
The script is the authority for this page. If a limit here disagrees with the file, the file is right.
- CODEcapture-indexing-signals-2026-09-08.py, the published capture script, for the seven signals, the method string and every limit quoted above.
- SPECGoogle's robots meta tag reference, for what
noindexandgooglebotmeta elements do. - SPECGoogle's guidance on consolidating duplicate URLs, for why a canonical tag is a hint rather than an instruction.
No indexation, traffic, ranking or revenue outcome is claimed anywhere on this page. The script measures declarations, and this page explains them.
This page was built by the pipeline it sells
The capture script on this page is what the pipeline runs to check its own pages. It refuses a keyword already saturated by an AI Overview, it will not let a page state a number that code did not compute, and it blocks a page that only restates what already ranks. It runs on your machine, there is no hosted service and no account, and you pay once.
Sixty day refund. Nothing about traffic, rankings or indexation is promised here or anywhere else on this site.
Keep reading
The seven signals above are declarations. To see how they interact with crawl rules, read the robots txt checker. To resolve the canonical a single page declares, use the canonical tag checker. For language alternates, see the hreflang checker, and for the full crawl rule picture see the robots census.