Scrunch serves bot-friendly HTML and pre-rendered pages to AI crawlers via Agent Pages. The user’s CDN routes AI user agents to Agent Pages, which delivers AI-optimized content directly to AI crawlers.

Additional context: Content served via Agent Pages is clean, server-rendered HTML with no JavaScript dependency. This may include technical normalization, content restructuring, and content enrichment if used in conjunction with Site Optimization.

Example

For example, a Scrunch user using Cloudflare as their CDN would follow this automated workflow:

Initial request: AI user agent (e.g., ChatGPT-User, PerplexityBot, etc.) retrieves webpage using the existing URL already indexed in search.

Routing logic: Cloudflare routes AI agent to the Agent Pages origin.

Optimized delivery: Agent Pages serves the agent an optimized version of the page designed for AI consumption.

Result: AI agents receive structured, easily parseable content while the human web visitor experience remains unchanged.

Follow-up question: Does Agent Pages change search indexing?

No, Agent Pages does not change search indexing. It only affects real-time AI retrieval bots, not search indexing bots. Search engine crawlers like Googlebot and Bingbot are not routed to Agent Pages.