Connect your data, where you want it, in milliseconds
Capture data in real-time from databases using CDC or streaming SaaS API's. Transform that data into views using stateful streaming SQL. Materialize views where you want them. Advance both streaming and batch data pipelines for fresh data and cost savings.
The Pitch:
🚰 Estuary Flow is a Real-Time Data Platform that enables building no-code reliable pipes that don’t require scheduling, and support batch/streaming and materialized views in milliseconds.
📒 A free account with up to 10gb/mo in data movement can be had here: www.estuary.dev
The Details:
Estuary Flow is built on top of an open-source streaming framework (Gazette) that combines millisecond-latency pub/sub with native persistence to cloud storage. Basically, it’s a real-time data lake.
Beyond being able to sync data continuously between sources/destinations without configuring, say, Kafka, there are a few benefits to a UI built on top of this streaming framework, specifically:
🗄️ Managed CDC. Simple, efficient change data capture from databases with minimal impact and latency. Seamless backfills – even over your very large tables that Debezium tends to choke on – and real-time streaming out of the box.
🧑💻 Streaming SQL transformations. We have a quite powerful transformation product that allows for Streaming SQL transforms without a requirement for windowing. Join historical data with real-time without having to think about it. Flow also offers schema validation and first-class support for testing transformations, with continuous integration whenever you make changes.
💽 Collections instead of Buffers. When a data source is captured – like Postgres CDC, or Kinesis, or streaming Salesforce – the data is stored in your cloud storage as regular JSON files. Later, you can materialize all of that juicy history and ongoing updates into a variety of different data systems. Create identical, up-to-date views of your data in multiple places, now or in the future.
📈 Continuous Views instead of Sinks. Materialized views update in-place. Go beyond append-only sinks to build real-time fact tables that update with your captured data – even in systems not designed for it, like PostgreSQL or Google Sheets. Make any database a “real time” database.
✅ Completely Incremental, Exactly-Once. Flow uses a continuous processing model, which propagates transactional data changes through your processing graph. This helps keep costs low while maintaining exact copies across different systems.
⏩ Turnkey batch and streaming connectors. Both real-time data as well as historical data supported through one tool and access to pre-built connectors to ~50 endpoints. For example, you can capture from the batch Stripe API, join it with data from Kafka and push that all to Google Sheets – all without building a custom integration. Or if you want, plug in your own connector through Flow’s open protocol.
About Estuary Flow on Product Hunt
“Connect your data, where you want it, in milliseconds”
Estuary Flow launched on Product Hunt on April 24th, 2023 and earned 227 upvotes and 115 comments, earning #3 Product of the Day. Capture data in real-time from databases using CDC or streaming SaaS API's. Transform that data into views using stateful streaming SQL. Materialize views where you want them. Advance both streaming and batch data pipelines for fresh data and cost savings.
On the analytics side, Estuary Flow competes within Data & Analytics, Data and Database — topics that collectively have 9.9k followers on Product Hunt. The dashboard above tracks how Estuary Flow performed against the three products that launched closest to it on the same day.
Who hunted Estuary Flow?
Estuary Flow was hunted by David Yaffe. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
David Yaffe