AI Access Policy

Policy details

Purpose

Conversion Encyclopedia is a technical reference built around structured, modular, and machine-readable conversion content.

Because the site receives substantial automated traffic from AI systems and search engines, this policy defines the conditions under which crawlers may access Conversion Encyclopedia.

Policy details

Support For Pay Per Crawl

Conversion Encyclopedia supports Cloudflare's Pay Per Crawl model and is preparing its access controls for paid AI crawling when the protocol is available for this zone.

After Pay Per Crawl is active, AI crawler access to site content must occur through authorized Cloudflare channels and may require compensation for successful crawl access.

Policy details

Unpaid AI Crawling

Unpaid AI crawling is not authorized as a permanent access model for training, enrichment, retrieval, or commercial AI use.

Until Pay Per Crawl enforcement is active, recognized official AI crawlers may temporarily access public content so semantic indexing, citations, and dataset refresh workflows can continue without interruption.

Policy details

Recognized AI Crawlers

The currently recognized AI crawler user agents include GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Amazonbot, Bytespider, TikTok Spider, and BaiduSpider.

These crawlers must identify themselves with a transparent user agent, respect reasonable request rates, avoid aggressive scraping, and must not impersonate another crawler or browser.

Policy details

Recognized Search Crawlers

Search indexing crawlers such as Googlebot, Bingbot, and Applebot are treated separately from AI crawler access because their primary role is discovery and search indexing.

Search crawlers must continue to identify themselves accurately, respect technical controls, and comply with published crawler guidance.

Policy details

Unknown Or Non-Compliant Crawlers

Unknown AI crawlers, incomplete user agents, empty user agents, generic scraping clients, and crawlers that imitate official services may be challenged, rate limited, blocked, or reviewed for further action.

Cloudflare logs, AI Crawl Control, bot verification signals, and traffic analytics may be used to distinguish official crawlers from spoofed or non-compliant traffic.

Policy details

Technical Requirements

Crawler operators must use a stable and transparent User-Agent header, provide a meaningful product identity, respect robots.txt, and avoid bypassing rate limits or security controls.

Automated clients must not use deceptive identities, rotate user agents to evade policy, overload origin resources, scrape private or restricted paths, or ignore Cloudflare access decisions.

Policy details

Future Enforcement

When Pay Per Crawl is active, unauthorized AI crawling may receive a payment-required response, a block response, a challenge, or another Cloudflare-managed enforcement action.

Conversion Encyclopedia may update this policy, Cloudflare rules, robots.txt references, crawler lists, rate limits, and access requirements as crawler behavior and Cloudflare capabilities evolve.

Policy details

Contact

Crawler operators, AI companies, and search providers can request access clarification, licensing discussions, or technical review by emailing ai-access@conversionencyclopedia.com.

Contact

Email Directory

AI access and crawler policy: ai-access@conversionencyclopedia.com