We are launching Crawlee for Python, an open-source library for web scraping and browser automation. Quickly scrape data, store it, and avoid getting blocked, headless browsers, and smart proxy rotation.
Hello Hunters and Makers,
I am Saurav, Developer Community Manager of Apify, the company building Crawlee.
I am happy to hunt Crawlee for Python today. We launched (Crawlee) in August 2022 and received an amazing response from the community, as well as continuous demand for building it in Python.
Finally, after a lot of hard work from our team, we are launching Crawlee for Python today.
It has all of these features:
- Unified interface for HTTP & headless browser crawling.
- Automatic parallel crawling based on available system resources.
- Written in Python with type hints - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
- Automatic retries on errors or when you’re getting blocked.
- Integrated proxy rotation and session management.
- Configurable request routing - direct URLs to the appropriate handlers.
- Persistent queue for URLs to crawl.
- Pluggable storage of both tabular data and files.
- Robust error handling.
Why use Crawlee rather than Scrapy?
- Crawlee has out-of-the-box support for headless browser crawling (Playwright).
- Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code.
- Complete type hint coverage.
- Based on standard Asyncio.
Please pass on your feedback and thoughts in the comments below!
About Crawlee for Python on Product Hunt
“Build reliable scrapers in Python”
Crawlee for Python launched on Product Hunt on July 9th, 2024 and earned 252 upvotes and 20 comments, placing #10 on the daily leaderboard. We are launching Crawlee for Python, an open-source library for web scraping and browser automation. Quickly scrape data, store it, and avoid getting blocked, headless browsers, and smart proxy rotation.
On the analytics side, Crawlee for Python competes within Open Source, Growth Hacking, Developer Tools and GitHub — topics that collectively have 770k followers on Product Hunt. The dashboard above tracks how Crawlee for Python performed against the three products that launched closest to it on the same day.
Who hunted Crawlee for Python?
Crawlee for Python was hunted by Saurav Jain. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Saurav Jain