Pay-per-Crawl

Pay-per-Crawl is a publisher monetization model in which AI companies pay a fee each time their bots fetch a page from a participating site. Instead of treating training and retrieval bots as free traffic, the publisher converts them into a paid API style relationship with metered access.

The model emerged in response to a sharp rise in AI and LLM bot traffic, which jumped from a small share of total web crawling to roughly 10 percent of all bot activity in less than a year. Vendors like TollBit, Cloudflare, Imperva, and Arc XP shipped Pay-per-Crawl products in early 2026 to formalise the relationship.

How it works

A publisher places a Pay-per-Crawl gateway in front of its site or its content APIs. The gateway identifies known AI bots through user agents, IP ranges, and signed identity headers, then either serves content with a recorded transaction, returns a soft block with pricing terms, or offers a structured AI Feed at a higher tier.

Pricing varies by use case. Training crawls usually cost more than retrieval crawls, and premium archives, editorial vertical content, or first party data can carry their own rate cards.

Why it matters

For publishers, Pay-per-Crawl turns an uncontrolled cost center into a measurable revenue line, with reporting on which AI products consume which content and at what price.

For AI companies, it provides a legal, predictable supply of high quality content and reduces the risk of rights disputes that have become common around large scale scraping.

Related terms: AI Feeds, Bot Traffic, Content Licensing, Publisher Monetization, RAG.