This product was not featured by Product Hunt yet. It will not be visible on their landing page and won't be ranked (cannot win product of the day regardless of upvotes).
Scrawly
Free open-source SEO crawler for Google and AI search
Scrawly is a free, open-source desktop SEO crawler for macOS, Windows and Linux. It crawls your site, renders JavaScript and runs 250 technical SEO, GEO and AI-search checks, then tells you in plain English what to fix. Unlike Screaming Frog or Sitebulb, there's no licence, no account and no page cap. It also tests whether GPTBot, ClaudeBot and PerplexityBot can reach you, audits llms.txt, applies reversible WordPress fixes and ships an MCP server. MIT licensed.
Hey Product Hunt 👋 I'm Ilias, a white label SEO consultant from Dhaka, Bangladesh.
For the last few years I've done technical SEO audits for agencies, mostly with paid desktop crawlers. They're excellent tools. But two things kept bugging me:
1. Every freelancer and junior SEO I mentored hit the same wall: a licence fee or a free tier capped at a few hundred URLs, right when they needed to learn on real sites.
2. My clients started asking a new question: "Can ChatGPT and Perplexity actually see us?" None of my tools answered it properly.
So I built the crawler I wanted, and I'm giving it away.
What Scrawly does:
• Crawls any site and renders JavaScript in real Chromium (desktop and Googlebot Smartphone)
• Runs 250 checks across 23 areas: indexability, redirects, canonicals, titles by pixel width, structured data, Core Web Vitals, hreflang, security and more
• Tests 10 crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended…) against robots.txt AND with a live fetch, so you catch the CDN or firewall rules that silently block AI bots
• Audits llms.txt and scores how ready your site is for AI agents, from Level 0 to 5
• Pulls in Search Console, GA4 and PageSpeed data, exports white-label HTML and PDF reports
• Fixes titles, meta descriptions, canonicals and redirects on WordPress, with a snapshot and an undo button for every change
How the approach changed: I started with a simple "list the problems" crawler. Real audits taught me that a list of 900 warnings helps nobody. So Scrawly now ranks findings by how many pages they hit, never reports the same problem twice, and gives every finding a fix you can paste into your coding agent.
It runs entirely on your computer. No account, no analytics, no page cap, MIT licensed. Install is one line in Terminal or PowerShell, and it keeps itself updated.
I'd love your honest feedback, especially:
→ Which check is missing from your audit workflow?
→ What would make you switch from your current crawler?
Thanks for checking it out 🙏
No comment highlights available yet. Please check back later!
About Scrawly on Product Hunt
“Free open-source SEO crawler for Google and AI search”
Scrawly was submitted on Product Hunt and earned 0 upvotes and 1 comments, placing #87 on the daily leaderboard. Scrawly is a free, open-source desktop SEO crawler for macOS, Windows and Linux. It crawls your site, renders JavaScript and runs 250 technical SEO, GEO and AI-search checks, then tells you in plain English what to fix. Unlike Screaming Frog or Sitebulb, there's no licence, no account and no page cap. It also tests whether GPTBot, ClaudeBot and PerplexityBot can reach you, audits llms.txt, applies reversible WordPress fixes and ships an MCP server. MIT licensed.
Scrawly was featured in Open Source (69k followers), SEO (38.3k followers), Developer Tools (521.5k followers) and GitHub (41.4k followers) on Product Hunt. Together, these topics include over 149.9k products, making this a competitive space to launch in.
Who hunted Scrawly?
Scrawly was hunted by Ilias Sami. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Want to see how Scrawly stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
Ilias Sami