Free Local AEO & SEO Spider and a Markdown content extractor
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Hey Product Hunt! I built crawler.sh because I kept running into the same problem: every time I needed to audit a website's SEO or extract its content, I had to choose between bloated enterprise tools, slow cloud services, or stitching together a bunch of scripts.
crawler.sh is a single tool that does all three: crawling, SEO analysis, and content extraction, from the terminal or a native desktop app. It's built in Rust so it's fast, it runs locally so your data stays private, and it outputs standard formats so you can take the results anywhere.
Hey Product Hunt! I built crawler.sh because I kept running into the same problem: every time I needed to audit a website's SEO or extract its content, I had to choose between bloated enterprise tools, slow cloud services, or stitching together a bunch of scripts.
crawler.sh is a single tool that does all three: crawling, SEO analysis, and content extraction, from the terminal or a native desktop app. It's built in Rust so it's fast, it runs locally so your data stays private, and it outputs standard formats so you can take the results anywhere.