Crawler reference / robots.txt
PerplexityBot and Perplexity-User: what is the difference?
Each user agent below has a separate job. The table uses the vendor's own wording. The robots.txt block follows the table, and every claim carries the vendor page it came from.
| User agent | Job | What the vendor says it does | Effect of blocking |
|---|---|---|---|
PerplexityBot | Search index | Surfaces and links websites in Perplexity search results | The site cannot be indexed for those results |
Perplexity-User | User action | Visits a page to answer a user question and links it in the response | The system cannot retrieve the page for user requests |
Neither bot is a training crawler
The Perplexity crawler page states that PerplexityBot is not used to crawl content for AI foundation models. It states the same for Perplexity-User, which is not used for web crawling or to collect content for training. The two tags cover search indexing and user-requested retrieval, not model training.
Why a user fetch ignores robots.txt
Perplexity documents that Perplexity-User generally ignores robots.txt rules, because a user requested the fetch. A Disallow line for Perplexity-User is therefore not a reliable block for that traffic. The tag exists so a site owner can express a preference and control which sites user requests can reach.
The one dependable control for a user-initiated fetch is the response itself: an authentication wall, a rate limit, or a server-side check of the request. A robots.txt line is a statement of intent, not an access control.
The robots.txt block
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /Both groups are already explicit on this site. The wildcard group also allows the site and blocks only /dl/. Perplexity says a robots.txt change can take up to 24 hours to reach its systems, so do not expect an instant effect.
How to check the result
Fetch a URL with the PerplexityBot string and confirm a 200. Use the robots.txt checker for the rule test, and read noindex vs robots.txt before you combine a block with a noindex tag.
Sources
Checked 2026-10-28 against the vendor page. Vendor wording can change; re-read the source before you copy a rule.
