This product was not featured by Product Hunt yet. It will not be visible on their landing page and won't be ranked (cannot win product of the day regardless of upvotes).
A robots.txt validator and URL testing tool that models Google’s documented parsing rules. Find blocking mistakes, ignored directives, sitemap issues, and crawler conflicts, then test the exact rule applied to any URL before publishing.
Hey Product Hunt,
I’m Logan Mosby, an SEO professional with more than 10 years of experience across technical, local, eCommerce, national, international, franchise, and enterprise SEO.
I built this robots.txt validator and testing tool because small mistakes in a robots.txt file can have major consequences, especially when a site is being redesigned, migrated, launched, or updated.
The file may look simple, but crawler behavior depends on user-agent groups, path matching, wildcard rules, precedence, and the exact directive that wins for a given URL. The tool is designed to make that logic easier to understand before a change is published.
You can use it to:
• Paste a live robots.txt file or an unpublished draft
• Analyze the file without fetching or uploading anything
• Identify accidental full-site blocks
• Find blocked CSS, JavaScript, and rendering resources
• Flag deprecated or unsupported directives such as noindex, crawl-delay, and host
• Check for missing sitemap references
• Review AI crawler access policies
• Test a specific URL against Googlebot, Bingbot, GPTBot, or a generic crawler
• See the exact Allow or Disallow rule that wins
• Review findings organized as Critical, Warning, Opportunity, or Passed
The parser follows Google’s publicly documented robots.txt behavior, including specific user-agent matching, wildcard paths, rule precedence, and Allow winning when matching rules have equal length.
There are a few important limitations.
A robots.txt file controls crawling, not guaranteed indexing. A blocked URL may still appear in search results if Google discovers it through links. Robots.txt is also publicly accessible and should never be used to protect private or sensitive content.
Other crawlers may interpret unsupported fields, wildcard behavior, or precedence differently. The tool therefore treats Google’s documented behavior as the primary model while still letting users test policies for other named crawlers.
Nothing entered into the analyzer is sent to a server or stored. The entire analysis runs in the browser.
I’d appreciate feedback on the parsing logic, severity classifications, and any additional robots.txt checks that would make the tool more useful during launches and migrations.
Catching robots.txt mistakes before publishing is such a practical tool for SEO folks — modeling Google's actual parsing rules is a nice touch. Congrats on the launch!
About Robots.txt Validator & Testing Tool on Product Hunt
“Validate robots.txt rules before you publish”
Robots.txt Validator & Testing Tool was submitted on Product Hunt and earned 6 upvotes and 2 comments, placing #66 on the daily leaderboard. A robots.txt validator and URL testing tool that models Google’s documented parsing rules. Find blocking mistakes, ignored directives, sitemap issues, and crawler conflicts, then test the exact rule applied to any URL before publishing.
Robots.txt Validator & Testing Tool was featured in Robots (10.7k followers), Marketing (467k followers) and SEO (38k followers) on Product Hunt. Together, these topics include over 93.4k products, making this a competitive space to launch in.
Who hunted Robots.txt Validator & Testing Tool?
Robots.txt Validator & Testing Tool was hunted by Logan Mosby. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Want to see how Robots.txt Validator & Testing Tool stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
Logan Mosby