Thomas Schranz ⛄️Check if ChatGPT, Claude, and Google can crawl your site in 10 seconds. Find and fix robots.txt issues blocking AI visibility.
Comment highlights
Ahh nice one! Just got the test done for AdAmigo.ai, and we are good to go! Nice launch - just upvoted :)
This is an excellent marketing move to get someone's email address! :D You are a marketing genius! :D :)
Been wondering if my site’s actually visible to all these new AI bots—this is such a smart shortcut! Does it also suggest exact fixes for robots.txt issues, or just flag them?
Lets me be the First Comment!
This is such a clever tool. As someone who’s been tweaking robots. txt files manually, this instantly checks AI crawler access—huge time-saver.
About AI Search Crawler Check on Product Hunt
“Check if your robots.txt allows ChatGPT, Google Gemini”
AI Search Crawler Check launched on Product Hunt on September 28th, 2025 and earned 180 upvotes and 8 comments, placing #6 on the daily leaderboard. Check if ChatGPT, Claude, and Google can crawl your site in 10 seconds. Find and fix robots.txt issues blocking AI visibility.
AI Search Crawler Check was featured in Analytics (171.4k followers), Marketing (462.7k followers) and SEO (37.5k followers) on Product Hunt. Together, these topics include over 91.4k products, making this a competitive space to launch in.
Who hunted AI Search Crawler Check?
AI Search Crawler Check was hunted by Thomas Schranz ⛄️. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Reviews
AI Search Crawler Check has received 1 review on Product Hunt with an average rating of 5.00/5. Read all reviews on Product Hunt.
Want to see how AI Search Crawler Check stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
TIL: Stackoverflow allows all crawlers (by accident)
I'd expect Stackoverflow to
✅ allow all search crawlers
🛑 block all LLM training material crawlers
when you check: all crawlers allowed (!)
but that's not the whole story
they do have a robots.txt w/ blocks (!)
the robots.txt says:
🛑 do not crawl any pages ("/")
it even says explicitly:
🛑 not for search
🛑 not for ai training
and it applies to any user agent, any crawler ("*")
but why does the crawler check then say everything is green?
the robots.txt is served w/ status 418
418 originated as a joke status code:
I'm a teapot
but that status code is in the 400-499 range (!)
and the RFC for robots.txt (9309)
says that if the robots.txt is served with a status in the 400-499 range
a crawler may access any resources
as if there is no robots.txt at all
and that's why crawler check currently turns up green for Stackoverflow
not sure what is right for Stackoverflow
nor if they have robots.txt set up like they intended
(I'd expect them to block crawlers for ai training but allow crawlers for search)
but it shows:
getting robots.txt right can be quite tricky