Product upvotes vs the next 3
Product comments vs the next 3
Product upvote speed vs the next 3
Product upvotes and comments
Product vs the next 3
AI Search Crawler Check
Check if your robots.txt allows ChatGPT, Google Gemini
Check if ChatGPT, Claude, and Google can crawl your site in 10 seconds. Find and fix robots.txt issues blocking AI visibility.
Top comment
About AI Search Crawler Check on Product Hunt
“Check if your robots.txt allows ChatGPT, Google Gemini”
AI Search Crawler Check launched on Product Hunt on September 28th, 2025 and earned 180 upvotes and 8 comments, placing #6 on the daily leaderboard. Check if ChatGPT, Claude, and Google can crawl your site in 10 seconds. Find and fix robots.txt issues blocking AI visibility.
On the analytics side, AI Search Crawler Check competes within Analytics, Marketing and SEO — topics that collectively have 671.6k followers on Product Hunt. The dashboard above tracks how AI Search Crawler Check performed against the three products that launched closest to it on the same day.
Who hunted AI Search Crawler Check?
AI Search Crawler Check was hunted by Thomas Schranz ⛄️. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Reviews
AI Search Crawler Check has received 1 review on Product Hunt with an average rating of 5.00/5. Read all reviews on Product Hunt.
For a complete overview of AI Search Crawler Check including community comment highlights and product details, visit the product overview.
TIL: Stackoverflow allows all crawlers (by accident)
I'd expect Stackoverflow to
✅ allow all search crawlers
🛑 block all LLM training material crawlers
when you check: all crawlers allowed (!)
but that's not the whole story
they do have a robots.txt w/ blocks (!)
the robots.txt says:
🛑 do not crawl any pages ("/")
it even says explicitly:
🛑 not for search
🛑 not for ai training
and it applies to any user agent, any crawler ("*")
but why does the crawler check then say everything is green?
the robots.txt is served w/ status 418
418 originated as a joke status code:
I'm a teapot
but that status code is in the 400-499 range (!)
and the RFC for robots.txt (9309)
says that if the robots.txt is served with a status in the 400-499 range
a crawler may access any resources
as if there is no robots.txt at all
and that's why crawler check currently turns up green for Stackoverflow
not sure what is right for Stackoverflow
nor if they have robots.txt set up like they intended
(I'd expect them to block crawlers for ai training but allow crawlers for search)
but it shows:
getting robots.txt right can be quite tricky