The AI Conversations Leaking Into Your Search Console via @sejournal, @suganthan
All Paid Media PPC News Social MediaAdvertising Video Advertising Columns Ask A PPC ExpertNEW PPC Pulse Rundown Why AI volume alone can't deliver personalization Your content architecture may be the real bottleneck. Get the Rundown on what needs to change in your content ops. Digital Marketing All Digital Marketing Analytics & Data Ecommerce Lead Generation Content Marketing Social Platforms Google YouTube Reddit LinkedIn TikTokNEW WordPress Other CMS Platforms Webinar New AI Search & SEO KPIs: 4 Real Signals AI mentions and citations are benchmarks, not decisions. Get 4 traffic-predictive signals drawn from real bot data across hundreds of sites. SEJ Live Webinars Rundowns PodcastNEW Ebooks All Resources SEJ Live AI Search Visibility for Local Business August 26 | How ChatGPT, Gemini, and AI Overviews choose a single local recommendation - plus reviews, GBP, and attribution. A classifier for the AI conversations leaking into Search Console. Seven kinds, from human replies and comparison questions to tracker bots and agent prompts. Suganthan Mohanadasan 4 hours ago ⋅ 20 min read Suganthan Mohanadasan Co-founder at Snippet Digital Bio Follow In early August, an SEO called Anastasia Kourou noticed queries in her Search Console report that didn’t look like searches: “Yes.” “Yes go on.” “Yes, pricing.” She posted a screenshot and asked John Mueller whether Search Console was tracking what people say to AI. Search Console includes AI Overviews and AI Mode data in the general performance report, and Google’s documentation explains the mechanism. A follow-up question inside AI Mode counts as a brand new query, and everything in the response gets attributed to it. When someone tells the AI “yes go on” and your page appears in what comes back, Search Console records an impression for your page against the query “yes go on.” Search Engine Roundtable covered the thread on August 6, and Ross Tavendale asked the question everyone was circling. If Search Console is recording people’s responses to AI Mode, how do we reverse engineer it? I pulled every one of these fragments out of 16 months of my own Search Console data, worked out that they come in seven recognizable kinds, and built the classifier into my free Search Console MCP so you can run it on your site with one prompt. Sorting them is what makes the leak usable. I found 1,127 queries and 20,300 impressions across 16 months on my site, against millions of ordinary impressions. Small, and every row of it is something a real session said or ran. Google launched Generative AI performance reports in June and, as of August 11, they’re live for everyone. The report shows how often your pages appear inside AI Overviews and AI Mode. It has five data views. Impressions, pages, countries, devices, dates. Queries and clicks are the two things it leaves out, and there’s no API support either. I checked that last part on my own property just to be sure. The Search Analytics API’s type parameter still ends at googleNews, the searchAppearance dimension returns nothing AI-related, and the BigQuery bulk export schema doesn’t have an AI column. The export button in the UI is the only way this data leaves Google. So, the report tells you how much AI visibility you have and refuses to say for what. (I think we all know why lol) Meanwhile the ordinary performance report, the one you’ve been reading for years, has been picking up AI conversation fragments the whole time. Nobody filters them out because officially they’re just queries. AI Mode looks like a chatbot. Underneath, every message is processed as a Google search, including the follow-ups, and Google folds all of it into the web search type alongside the classic 10 blue links. Your query report now holds two different things. Searches people typed, and fragments of conversations people had with a model that happened to show your page. The position data proves it’s the second thing. My site shows average position 4.5 for the query “yes.” On the open web, that ranking is impossible; “yes” belongs to songs and grammar sites. Inside an AI response, it makes sense, because Google’s documentation says links in an AI Overview inherit the position of the whole block, and AI Mode citations get counted under the same rules once they scroll into view. Position 4.5 on a reply word means my link sat inside the answer block, not on a results page. That’s the leak. The question is whether the fragments can be told apart from normal queries at scale. I pulled 16 months of queries from my property and classified everything that couldn’t be a typed search. Seven kinds came out, and each one has a different origin. Note: I haven’t included all of my queries for obvious reasons, just a sample. When you run this on your own GSC account, you’ll get all of your data. Bare replies: “yes,” “sure,” “really?,” “show me.” A person answered the AI mid-conversation; the reply was processed as a search, and your page appeared in the response. Positions here come from the answer block, not a results page. Mid-conversation comparisons: “what about resend?,” “what about gemini,” “how about in chinese?” The person has an answer in front of them and asks the AI to test an alternative, and the alternative they name is the one they care about. Questions addressed to someone rather than typed at a search box: “can you jailbreak meta raybans,” “how do i sell it,” “is it free.” The giveaway is grammar that only works with a listener, which gets its own section below. Synthetic prompts from AI visibility tools, run on a schedule. Two signatures in my data. Prompts ending “. my location is usa.” and prompts in the form “evaluate the [company] on [facet].” Nobody searches the same sentence every day for two months, so these are software. A machine’s complete instructions, logged whole: “search the web for… return the 3 most relevant results you actually found … do not invent results or urls.” Somewhere an engineer wrote a prompt template, and Google filed it as a query. Error messages and spreadsheet headers searched as-is, by people or pipelines. My data includes a rank tracker’s full CSV column header as a single query. Ten or more words with no other marker. Some are quoted sentences, some are agents, some are people. The classifier files them for review instead of guessing, which is the right amount of confidence for this bucket. The classifier is a ladder of checks, applied in order, and a query stops at the first one it matches. Take “what about resend?” It isn’t on the reply list, so it passes the first rung. It matches the second, a pivot, “what about” followed by a short noun. Classification done, and the row tells me the rest. There’s no machine learning in any of this. The patterns are a curated list anyone can read and disagree with, which is the point. Every classification is explainable. The obvious objection to all of this is that long queries existed before AI. [how to get not provided keywords in google analytics] is nine words, and nobody said it to a chatbot. The split comes down to who the query is addressed to. When I first showed this to my co-founder Andy, that was his question. A long-tail query is a detailed request addressed to nobody. It has its own subject and names its own tools. A conversational query is addressed to someone, and four signals give it away. Any of those four, at any length, and the query is conversational. Ten or more words with question syntax gets classified as conversational, because typed queries average two to four words and almost nobody types 11. Ten or more words with no other marker goes to the review pile instead of into a claim. And anything at nine words or fewer with none of the signals is treated as an ordinary search, which is why the not provided query above never enters the dataset.
Source: Search Engine Journal
This article has been carefully curated and reformatted for educational and informational purposes. Full credit goes to the original publisher.
📚 Visit more helpful articles on Joab Peters Blog
No comments