AI bots like ChatGPT, Claude, and Perplexity are now crawling the web to feed their models - but how do they actually behave?
Do they follow crawl budgets like Googlebot? What makes them visit some pages and ignore others?
In this talk, Serge Bezborodov shares insights from analyzing billions of log lines across real websites.
You'll see what AI bots really crawl, how often, and why.
This is one of the first large-scale studies showing how these new bots behave in the wild—and what it means for your website and content.
