Google Is Fighting SERP Tracking. AI Agents May Make It Worse via @sejournal, @martinibuster
All Paid Media PPC News Social MediaAdvertising Video Advertising Columns Ask A PPC ExpertNEW PPC Pulse Guide AI Max Invalid Traffic Doubled, Weeks Before Black Friday Invalid traffic on AI Max search doubled to 5.28%. Get the channel-by-channel retail benchmarks before Black Friday. Digital Marketing All Digital Marketing Analytics & Data Ecommerce Lead Generation Content Marketing Social Platforms Google YouTube Reddit LinkedIn TikTokNEW WordPress Other CMS Platforms Guide Your ChatGPT Calls Are Hiding in the 'Direct/Other' Bucket Calls from ChatGPT, Claude, and Perplexity land in 'direct/other.' This checklist shows six steps to track them as their own source. SEJ Live Webinars Rundowns PodcastNEW Ebooks All Resources Ebook State Of Search 2027 Where search leaders are putting 2027 budget, plus the AI visibility scorecard that replaces session reporting. SEJ ⋅ SEO Google Is Fighting SERP Tracking. AI Agents May Make It Worse Google is slamming the door on SERP tracking, but AI agents may be giving it more reason to shut down automated queries. SEJ STAFF Roger Montti 4 seconds ago ⋅ 5 min read SEJ STAFF Roger Montti Owner - Martinibuster.com at Martinibuster.com Bio Follow Google is entering a new phase of SERP scraping due to AI agents. Some of it may be driven by networks of LLMs that are reverse engineering Google’s search results. The public face of Google’s war against scraping remains rank trackers but the real battle may be elsewhere. Will the ability to track rankings become a thing of the past? What may be a reason behind Google’s actions to stop rank tracking is that the amount of AI Agent traffic has grown exponentially. According to Cloudflare, which is in a position to monitor web traffic, AI agent website traffic has increased 1,700% over the past year. “At the end of 2024, Cloudflare handled an average of 63 million HTTP requests a second. Today, it’s almost doubled to 115 million, with peaks above 150 million. Over the past year, daily requests from AI agents on our network grew by more than 1,700%. This year, for the first time, more than half of Internet traffic wasn’t human.” The problem with AI agent web traffic is that some of it a hybrid human/agent visit, agents working on behalf of humans, which means that some of it still has value to websites, something Google may have already considered in formulating engineering responses to rank trackers. The problem for Google is that a massive amount of traffic could cause latency issues in serving search results, including AI search results. This is a bigger problem for keyword that trigger AI-driven search results because those are computationally more expensive to generate. Regular search ranks pre-indexed websites that are associated with various queries. AI-derived search results include LLM inference over multiple queries, which means that every time a user asks one question, the LLM generates multiple associated queries, plus various reasoning steps, before answering with a synthesized summary of all the sources. AI Mode is inherently more computationally expensive than traditional Search, which is why Google is aggressively reducing its AI cost per query as usage scales through improvements in LLMs, hardware and engineering optimizations. Google’s Sundar Pichai remarked multiple times on Google’s efforts to drive the search per query costs down in the Q2 Earnings call. The cost of AI queries was so important that it merited multiple mentions: “Yesterday we announced new models — Gemini 3.6 Flash and Gemini 3.5 Flash-Lite — which are cost effective and highly efficient. We’re seeing tons of demand for our workhorse Gemini Flash series, because it hits the sweet spot of performance and cost. As we serve more of these queries, we’ve continued to drive efficiencies. Thanks to our engineering and hardware optimizations, this quarter we reduced the cost of AI Mode responses to its lowest level since launch, even as we’ve brought more advanced AI capabilities.” The high cost per query of AI-driven Search is a real problem. AI Search queries are computationally more expensive to serve and AI agent traffic can effect search results latency. Google may very well have taken a more adversarial approach to automated rank tracking in order to improve the cost per query calculation. Google has published multiple research papers focused on AI-generated content. The papers allude to attackers changing their content in response to Google’s reactions in order to unlock rankings. Core to unlocking rankings is tracking Google’s search results. Although the research papers don’t mention distillation, the process of reverse engineering an LLM through the outputs, the few details the research paper shares imply that is what Google is really fighting against. Ryan Jones (LinkedIn profile), developer of the SERPrecon service, extracts competitor keyword and historical search data that informs their keyword analysis for content optimization. As the principal behind this tool he is an authoritative source of information about rank tracking. “we’re getting to the point where nobody will have rank tracking anymore because of the crazy amount of AI scraping going on.” Jones offered the following response to someone’s tongue-in-cheek suggestion to us AI to scrape the search results (SERPs): “that’s why the rank trackers are failing. Google is blocking all the AI trackers AND AI itself… and the rank trackers are getting caught up” Something that has not been mentioned by Google and is not frequently discussed is that a reason why Google is discouraging rank tracking is that the data can be used to understand Google’s ranking output to build an AI Model that can reproduce Google’s search results through a process called distillation. It may very well be that Google’s adversarial anti-rank tracking actions may be primarily an attempt to stop companies from reverse engineering Google’s search results. Google’s decisions are made based on data and one of the problems with rank trackers is the increased cost per query. Another issue is that rank trackers distort the actual keyword inventory data. The inflation of keyword inventory is a long-running problem that predates the AI agent era, going back to the earliest days of SEO. But what’s happening today is a massive increase that quite likely distorts Google’s click and traffic metrics. AI-driven automated crawling on behalf of humans and AI search has surged across the web. Google has to respond to automated search results scraping. Ryan Jones says the two trends are connected. Add SEJ as a preferred source on Google Read Full Bio SEJ STAFF Roger Montti Owner - Martinibuster.com at Martinibuster.com I have 25 years hands-on experience in SEO, evolving along with the search engines by keeping up with the latest ... Breaking news and insights to help you lead search strategies. Why Scaled AI Content Fails: Google’s Crawl Economics Explained Charging AI Bots Decides Which Agents Can Still Cite You How AI Agents Decide Which Brands To Recommend: Trust Is The New Ranking Factor The AI Search & SEO News That Matters Breaking news and insights to help you lead search strategies. Breaking news and insights to help you lead search strategies. In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses -- and careers.
Source: Search Engine Journal
This article has been carefully curated and reformatted for educational and informational purposes. Full credit goes to the original publisher.
📚 Visit more helpful articles on Joab Peters Blog
No comments