Every technical SEO audit run in the past decade was built around a single consumer: Googlebot. There were other crawlers on the list, Yandex, Bingbot, Yahoo, but Googlebot set the rules. Crawlability, indexability, page speed, schema markup. Everything else followed.
That model is now incomplete. Roughly 30 percent of all web traffic is bots, and the AI crawlers from OpenAI, Anthropic, Perplexity, and Microsoft are arriving with different technical requirements, different rendering capabilities, and a very different tolerance for slow or complex pages.
On this episode of SEOTalk Spaces asked one question: does the audit template most practitioners have used for a decade pass the AI readiness test?
The panel’s answer was clear. The existing template is not wrong. It just optimises for one bot out of a dozen.
Key takeaways
- Four of six major AI crawlers cannot render JavaScript — client-side rendered sites are invisible to most of them
- Log file analysis is no longer optional: 499 status codes show exactly where AI bots gave up on your site
- LLMs.txt remains unproven in practice — implement it quickly if asked, but do not let it dominate the conversation
- Cannibalization is the most common root cause of unexplained traffic loss and one of the most under-diagnosed issues in a standard audit
- JSON-LD structured data gives AI systems a fast path to understanding your page before they commit to processing the full DOM
The Rendering Problem No One Is Checking For
Parth opened with a detail that should concern anyone managing a JavaScript-heavy website. Four of the six major AI crawlers cannot render JavaScript at all. When those bots arrive at a page where the content loads client-side, they see a blank page. They cannot read the content. They cannot perform chunk retrieval. They move on.
This is not a theoretical risk. It is the default state for a large share of modern websites built on React, Vue, or Angular where server-side rendering was never a priority because Googlebot eventually handles client-side rendering reasonably well. AI bots do not.
The fix is not necessarily a full rebuild. Suresh pointed to Cloudflare’s content negotiation feature, which can automatically serve a markdown version of a page to bots that request it. That strips away scripts and stylesheets and gives the bot clean text to process without requiring any changes to the underlying site. For sites not on Cloudflare, the path is server-side rendering, and it needs to be fast: Suresh’s threshold was time-to-first-byte under 100 milliseconds and total page load under one second.
What to do
Check whether your key pages are server-side or client-side rendered. If you are on Cloudflare, enable content negotiation so bots receive a markdown version of the page. If not, prioritise server-side rendering for your highest-value pages first.
Log Files Are Now a Mandatory Audit Step
Parth drew a clear line between what was optional before and what is required now. Log file analysis was always on the advanced checklist, the kind of step a practitioner might skip unless they had a specific crawl problem. That has changed.
Users hit website pages. AI bots hit servers. The log file is the only place where that server-level activity is recorded, and it contains signals that no dashboard or crawl tool surfaces.
The most actionable signal is the 499 status code: a client abort, meaning the bot sent a request and gave up before the server responded. Suresh tracks these closely across his clients and uses them as a direct measure of AI crawl efficiency. A site with frequent 499s is losing retrieval opportunities not because of content quality or entity authority, but because the server is too slow. The bot moves to the next source.
The log file also tells you which AI bots are visiting, how frequently, and which pages they prioritise. Cloudflare’s Q1 2026 data shared in the session showed ClaudeBot crawling approximately 20,600 pages per referral sent back, with OpenAI at around 1,300 and Meta at zero. Publishers considering whether to block AI crawlers can use that ratio as a starting point for the commercial conversation. SaaS brands chasing citations have a different calculation than publishers trying to protect content assets.
Should You Block AI Crawlers?
Malhar raised the question directly: if AI bots crawl thousands of pages and send almost no traffic back, is blocking them the right call?
Parth’s answer was grounded in what the brand is trying to do. Brands selling attention or protecting proprietary data have a reasonable case for selective blocking. Brands building for AI search visibility do not, because blocking a crawler that cannot cite you is the same as not existing for that platform’s users.
The more nuanced point: allowing crawls is not a passive decision. It is an active one that should be paired with log file monitoring, rendering checks, and speed optimisation. Letting GPTBot in means nothing if the pages it arrives at are client-side rendered, slow to respond, or return 499s.
Watch out
Allowing AI crawlers is not enough on its own. If your pages are slow, JavaScript-rendered, or return 499 status codes, the bot will arrive and leave without processing your content. Check the log files to confirm what actually happened after the crawl.
LLMs.txt: Real Signal or Just Noise?
Malhar called it the most debatable topic of 2026, and Parth’s answer was more measured than most of the commentary around it.
He has implemented LLMs.txt across three clients and seen no measurable improvement in any of them. His practical guidance: if a client asks about it, implement it because it takes 30 minutes and the downside is zero. But do not let it become the headline of an AI readiness conversation. The value of the conversation with a client is in rendering, log files, structured data, and entity authority, not in a text file in the root directory that most AI systems have not committed to using consistently.
The broader point applies to any tactic that gains momentum before the evidence catches up. Do it if it costs nothing. Do not anchor the strategic conversation to it.
What Structured Data Actually Does for AI Systems
Suresh brought a useful framework for thinking about JSON-LD that goes beyond the usual “it helps Google understand your page” explanation.
AI systems process pages in phases. In the first pass, lightweight models scan the page for basic signals: what the page is about, what kind of content it contains, what entities are present. JSON-LD structured data gives those lightweight models a fast, clean signal in a format they can process without parsing the full HTML. It is not a crawl directive. It is a context shortcut.
Once a lightweight model has enough confidence in the page’s relevance and quality, it passes the page to a more capable model for full processing. Passage indexing, internal link evaluation, and entity relationship mapping happen in that second phase. Pages with clean, well-structured JSON-LD move through the first phase faster and more reliably, which increases the probability of reaching the second phase at all.
The practical implication: structured data is not just a featured snippet tactic. It is an efficiency layer that helps AI systems decide whether your content is worth processing at depth.
The Case for Adding Cannibalization to Every Audit
David brought a different angle that added real weight to the session. In his view, the majority of sites he encounters with unexplained traffic loss are not victims of algorithm updates or AI Overviews. They are victims of cannibalization they have never diagnosed.
The pattern he described is familiar: a site grows content over time, often with location pages or topic variations built on similar URL structures, and multiple pages end up competing for the same queries. Traffic that used to concentrate on one strong page gets diluted across three or four weaker ones, none of which rank well. The site looks like it has a penalty. It actually has a structural problem.
Most standard technical audits do not include a cannibalization check as a named step, and David’s estimate was that around 80 percent of sites he reviews have it to some degree. Adding a cannibalization audit to the standard checklist, checking for partial-match slug overlaps and ranking collisions at the query level, would resolve more traffic problems than most of the items currently on the list.
Three Things to Add to Your Next Audit
Malhar closed the session by asking each panellist for one concrete addition to the standard audit template. The three answers were specific enough to act on.
Parth: check rendering. Confirm whether your key pages are server-side or client-side rendered, and verify that content inside accordions or expandable sections is not hidden from bots. A rendering check should be a named step, not an assumption.
David: run a cannibalization audit. Check for slug overlaps and competing pages before concluding that a traffic drop has any other cause.
Suresh: clean the head section. A bloated head section with too many scripts, render-blocking resources, and redundant tags slows the initial parse and reduces the probability that a lightweight AI model completes its first-pass evaluation. Sitewide wins in the head section compound across every page.
The underlying principle across all three: the mechanics of technical SEO have not changed. Crawlability, rendering, structured data, and site architecture are still the foundation. What has changed is the number and variety of consumers those mechanics need to serve.
What would you choose?
Join the conversation
Have you added an AI readiness section to your technical SEO audit? Which check surfaced the most surprising finding — rendering, log files, or something else entirely?
Frequently Asked Questions
What is the difference between a traditional technical SEO audit and an AI readiness audit?
A traditional technical SEO audit focuses primarily on Googlebot: crawlability, indexability, page speed, and schema markup for search engine ranking. An AI readiness audit adds a layer for the AI crawlers from OpenAI, Anthropic, Perplexity, and Microsoft, which have different rendering requirements, speed tolerances, and crawl patterns. The existing audit is not wrong; it is incomplete for a web where roughly 30 percent of traffic now comes from bots beyond Googlebot.
Why does JavaScript rendering matter so much for AI crawlers?
Four of the six major AI crawlers cannot render JavaScript. If a website loads its content client-side through a JavaScript framework, those bots see a blank page. They cannot read the content, perform chunk retrieval, or use the page as a citation source. Ensuring key pages are server-side rendered, or using Cloudflare content negotiation to serve a markdown version to bots, closes this gap without requiring a full site rebuild.
What are 499 status codes and why do they matter for AI search?
A 499 status code is a client abort: the bot sent a request and gave up before the server responded. In log files, frequent 499s are a direct signal that a site is too slow for AI crawlers to process reliably. When a bot cannot retrieve a page within its crawl budget, it moves to another source. Reducing 499s through server-side rendering and faster time-to-first-byte improves the probability that AI systems process and potentially cite your content.
Does LLMs.txt actually help with AI search visibility?
The evidence so far is limited. Parth has implemented it across three clients and seen no measurable improvement in any of them. The practical guidance is to implement it because it takes under 30 minutes and costs nothing, but not to anchor an AI search strategy around it. More impactful technical investments include server-side rendering, log file monitoring, structured data, and entity authority across third-party sources.
How does JSON-LD structured data help with AI retrieval specifically?
AI systems process pages in phases, starting with lightweight models that scan for basic signals. JSON-LD gives those lightweight models a fast, clean summary of what the page is about and what entities it contains, without requiring them to parse the full HTML or execute scripts. Pages with well-structured JSON-LD move through that first phase faster, which increases the probability of reaching the deeper processing phase where passage indexing and entity relationship mapping occur.
Is cannibalization still worth auditing in an AI search environment?
Yes, and possibly more so than before. Cannibalization fragments a site’s topical authority across multiple competing pages, weakening the signal any individual page sends to both traditional and AI-powered search systems. David’s estimate was that around 80 percent of sites with unexplained traffic loss have an undiagnosed cannibalization problem. A cannibalization check, looking for partial-match URL overlaps and competing pages ranking for the same queries, should be a named step in every standard audit.
Check the audio recap of the episode: