Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is actively altering its user agent, IPs, and ASNs to bypass websites' no-crawl preferences, disregarding explicit anti-crawling requests.
MAIN POINTS
- Perplexity modifies user agents to conceal its crawling activities.
- Changes in IPs and ASNs are used to evade detection.
- Websites' no-crawl preferences are being ignored.
- The actions are in direct conflict with site owners' explicit requests.
TAKEAWAYS
- Ethical web crawling requires respecting websites' no-crawl directives.
- Altering user agents and IPs can undermine trust with website owners.
- Transparency in crawling activities is crucial for maintaining web integrity.
- Websites may need stronger measures to enforce no-crawl preferences.