Check if ChatGPT, Claude, and Google can crawl your site in 10 seconds. Find and fix robots.txt issues blocking AI visibility.
Top comment
LLM SEO Index let's you check if your website(s) are visible to AI search assistants like ChatGPT, Perplexity, Claude as well as to traditional search engines like Google and Bing.
I recently noticed that the robots.txt for one of my websites actually stopped ChatGPT from crawling the website. Why? A few days ago I accidentally enabled a bot crawling feature on a web hosting platform that was configured stricter than I expected.
That's why we built LLM SEO Index, to easily identify problems like these.We currently cover:
ChatGPT
Perplexity
Claude
Amazon (Alexa)
Apple (Siri)
Meta AI
and
Microsoft Bing
DuckDuckGo
Yandex
Brave
With more and more users going directly to ChatGPT to get answers and recommendations it matters a lot whether AI search assistants can crawl your websites or not.
Unfortunately many websites block ChatGPT and other bots by accident. Serving of websites has become more and more complex in the last few days and often many different systems have to play well together to make sure your web properties are actually crawlable the way you want them to be crawlable.
With LLM SEO Index we built a simple check that you can run for your website(s) or for websites of your clients to get a fast high level result that helps you to dig deeper if there is anything unexpected going on.
We want to make this as good as it possibly can be. Any feedback you have is very welcome!
Comment highlights
Helpful mini-product! It's funny how publishers have been trying to 'keep the bots away' for years and now we all depend on them.
With every SEO decision, I find myself asking, "Will ChatGPT be able to take this information and relay it to others?" With this tool, I don't have to wonder anymore. It is a perfect solution to a common problem for online businesses.
Given it's really just a tool to check robots.txt you've done a great job at making it look like it's doing more than that.
The bizarre "Log in to get even more results" when you're kind of shown none at the start is a bit of a weird surprise, but I guess there isn't that much it's offering, so you can't let the cat out of the bag too early?
Hey @__tosh folks, Interesting product. Success for the launch.
I'd suggest to add clear recommendations moving forward. For instance, for my site https://uxarmy.com/ it doesn't show the impact and the action related to blocking of "ChatGPT (Training)".
"ChatGPT block detected (1/3 bots blocked)Some ChatGPT features may not work properly
ChatGPT (Training)"
A crucial tool! Congrats on the launch @__tosh @peterbuch
When I tried to login, it asks for extra permissions that might be confusing for the users as that usually appear. While its nothing extraordinary, but you should look into it as that might affect conversions.
Some top publishers blocking AI and SEO bots signals a strategic shift in controlling content access. As AI-driven search grows, website owners must rethink their crawling policies to capture new traffic sources beyond Google. An awesome tool as expected. Carry on guys.
Nice job team, being able to quickly check if ChatGPT, Claude, and Google can crawl a site in just 10 seconds is super handy for spotting robots.txt issues!
That's a great tool idea, I noticed a few weeks ago I was inadvertently blocking ChatGPT. Would be curious to see the full report from llmseomonitor, but once I am signed in, I'm encountering an error trying to check my website.