Web traffic patterns have shifted sharply. Machines now account for 35% of visits across thousands of sites tracked by Known Agents, down just 2% from the prior quarter. That share reflects a broader move toward automated systems handling discovery and interaction once left to people.
Data from 5,000-plus websites using their analytics tools shows clear trends. Agentrification, the portion of activity tied to autonomous AI agents, reached 28% and climbed 11% over the same period. Referrals from AI chat platforms remain small at 0.1% but dipped 9%. Bots comply with robots.txt directives 98.5% of the time, yet operators still face selective scraping.
Search engine crawlers lead all categories at 22.8% of traffic, followed by SEO crawlers at 19.7%. AI search crawlers contribute 12.4%. Top individual agents include bingbot at 8.2% and Googlebot at 7.9%. Operators from Google, Microsoft, and Ahrefs dominate the list.
Scraping for AI training stands out as a distinct stream. ClaudeBot leads that activity at 27%, with Meta and Amazon bots close behind. Law, government, and community sites see the heaviest scraping. Fetching for real-time AI responses falls mostly to OpenAI tools, which drive 86.4% of such volume.
Enterprise operators now weigh blocking strategies alongside visibility tools. Automatic robots.txt generation and targeted alerts help site owners manage exposure without blanket restrictions. Compliance rates stay high, but undocumented agents still slip through at low but persistent levels.
Recent industry roundups highlight parallel growth in agentic products. CRN reported on top tools from Cisco, Nutanix, and Microsoft in mid-2026, underscoring how these systems move from pilots to production workloads. Google Cloud’s trends report from the same year points to executives prioritizing orchestration and measurable returns.
Traffic composition will keep evolving. Site owners tracking these flows gain an edge in content placement and access controls. Known Agents data supplies the granular view needed to act on the numbers rather than guess at them.