Common Crawl Published A Manual For Being Visible To AI, I Automated It via @sejournal, @suganthan

Webinar How Freshpet Earned AI's Trust The GEO playbook behind Freshpet's AI Overview citations, plus checks to run on your own brand. Free, live Aug 20. Webinar Google Local Services Ads Are Moving To PMax: What To Check First Prepare for Google's LSA move into Performance Max with a before-and-after audit checklist from CallRail. Webinar SEJ Live: The AI Search Playbook June 17 | AI visibility, social citation signals, and multi-channel attribution — three sessions, live Q&A. Webinar How Freshpet Earned AI's Trust The GEO playbook behind Freshpet's AI Overview citations, plus checks to run on your own brand. Free, live Aug 20. Webinar New AI Search & SEO KPIs: 4 Real Signals AI mentions and citations are benchmarks, not decisions. Get 4 traffic-predictive signals drawn from real bot data across hundreds of sites. 🔥[Live 8/12 with Loren Baker] Ecommerce SEO: Own your "brand +promo code" search. Your CDN may be blocking CCBot without telling you, keeping your pages out of the archive that trains most models. Suganthan Mohanadasan 12 hours ago ⋅ 9 min read Suganthan Mohanadasan Co-founder at Snippet Digital Bio Follow 161 READS Every month, a non-profit called Common Crawl fetches over 2 billion webpages and gives the copy away to anyone who wants it. When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was roughly 60% filtered Common Crawl by training weight. If a model already knows your brand without searching the web, odds are this is part of where it learned. The crawler behind it is called CCBot, and whether it can read your site decides whether you exist in that archive at all. CCBot/2.0 (https://commoncrawl.org/faq/) The July 2026 crawl holds 2.14 billion of those pages. Three of its records come from bbc.com. CCBot asked for permission, read the refusal, and left. As far as the training data pipeline is concerned, one of the biggest news sites on earth doesn’t exist. The BBC robots.txt blocks 13 of the 14 AI crawlers the checker tracks, and other top news sites have been blocking training crawlers on purpose since 2023. BuzzStream measured it at 75% of the top U.S. and UK publishers. The question that interests me is how many sites are missing from that crawl without ever deciding to be. Chris Green posted HTTP Archive numbers on LinkedIn recently. Roughly 492,000 sites name CCBot in robots.txt, and about 95% of those mentions exist to block it. Since July 1 2025, Cloudflare blocks AI crawlers by default on every new domain it serves. Its managed robots.txt reached 3.8 million more domains that never wrote one. Ad plugins add blocklists. In 2026, a CCBot block is as likely to be a platform default as a decision. Common Crawl itself wrote a manual for auditing exactly this. But it’s all manual work. So I automated it. The Common Crawl Visibility Checker is free, needs no signup, and answers one domain at a time, straight from Common Crawl’s own archives. The crawling has run monthly since 2008, and the snapshots are the raw material for C4, RefinedWeb, RedPajama, Dolma, FineWeb and most of the other big training sets. Labs rarely touch the raw crawl. They take a filtered derivative, so your page’s route into a model runs from CCBot through Common Crawl into one of those datasets, and each step keeps only what passed the one before it. The pipeline is still moving, too. FineWeb added six fresh 2025 crawls last July, and NVIDIA hosts its Nemotron training set inside Common Crawl’s own bucket. I covered the pre-training life of your content in the Three Lives of schema markup. The short version is that CCBot is the one crawler whose visits compound. GPTBot fetches your page for OpenAI alone. A CCBot visit ends up with everyone who’ll ever train on the open web. The AI Visibility Audit is an 18-page guide telling site owners how to check their standing in the dataset. You curl your homepage with CCBot’s user agent and compare the response to a browser’s. Next comes the index API, where you eyeball your capture counts, and their web graph, where you look up your rank. Then you compile a scorecard per domain, by hand. lol! I read the checklist and then automated it. Conveniently, Common Crawl opened its data to browser tools in May, which makes this the rare tool the data provider actually wants built. I think they know exactly what they’re doing. Fair play to them. How many of your pages each of the last twelve monthly crawls took, as a trend. My site went from two captured pages in last August’s crawl to 55 in July’s, which is the most flattering chart anyone has ever drawn of it. A big domain gets an estimate instead. Type wikipedia.org and you’ll see roughly 3.7 million, labelled as an estimate, because the tool reads the index structure for huge sites rather than counting every record.

---

Source: Search Engine Journal

This article has been carefully curated and reformatted for educational and informational purposes. Full credit goes to the original publisher.

---

📚 Visit more helpful articles on Joab Peters Blog