Data observability that drives action. Detect and resolve data quality incidents faster by viewing data flowing through your systems and acting on them in real-time with Streamdal, the open-source data observability tool.
We're so excited to share the next evolution of Streamdal with PH! Thanks @mwseibel for Hunting us.
Streamdal is an open-source data observability tool that drives action. Data engineering teams will be able to detect and resolve data incidents faster by viewing data flowing through their systems and acting on it in real-time.
We built Streamdal because we've experienced a growing divide between the systems that handle data and the tooling used to detect and resolve incidents. Observing data happens downstream once data is already in data stores, while at the same time detecting and resolving data incidents typically requires you to trace the issue back upstream.
We believe both observability, detection, and resolution should exist upstream.
Streamdal helps teams overcome this divide by generating a real-time, dynamic visualization of the data flow: we call it “The Data Graph”. This means teams will get a dynamic bird's-eye view of their producers and consumers as they scale up and down in real-time. From there, they’ll be able to watch a live view of data flowing throughout their system.
By importing and wrapping a few lines of code with the Streamdal SDK, they’ll be able to detect data incidents the moment they happen. Along with the dynamic data graph, they’ll get throughput metrics, schema inference, and the ability to zoom in and observe real-time data - it's essentially a `tail -f`, but for data and in human readable format.
Today, we're excited to share our open-source product with the PH community! In this beta phase of our launch, we currently have SDK support for Golang, Python and Node.js. The UI and server components can be deployed in containers anywhere: on-prem or in cloud.
Coming very soon is a host of data quality management features allowing you to proactively interact with real-time data, and prevent downstream issues! Think of it as a `firewall` for your data.
We'd love to hear your thoughts, ideas, and feedback. We'll be here to answer questions in the comments.
Wow, this looks so cool!
You had me already at just the beautiful UI...
And the capabilities... Man
A very very smart idea. Is there a learning feature that will increasingly automate dq corrections?
Congratulations Team Streamdal for your impactful launch on Producthunt! Your open-source tool could revolutionize the way businesses strategize around their data. Having a real-time dashboard to track data incidents is absolutely genius. I do have one suggestion that could help further elevate your tool - consider implementing predictive analytics to anticipate potential data issues before they even occur. A system for alerts on possible future data incidents, might be an interesting and powerful feature worth exploring. Keep innovating!
Hello guys, congrats on your impressive launch!🎉
I‘m planning to create an entrepreneurial support group, because I've noticed that many “PH support groups” on #LinkedIn aren't actually helping each other👎, but at the same time, early-stage project teams all need:
1⃣️ help with marketing
2⃣️ Product testing & feedback
3⃣️ finding initial users
So, why not #makershelpmakers ?
Plz let me know if you are interested!
Congrats on your launch Y'all.
How much overhead does this add to something like a NodeJS script?
This is all that's needed?
import { OperationType, Streamdal } from "@streamdal/node-sdk/streamdal";
export const example = async () => {
const streamdal = new Streamdal({
streamdalUrl: "localhost:8082",
streamdalToken: "1234",
serviceName: "test-service-name",
pipelineTimeout: "100",
stepTimeout: "10",
});
const result = await streamdal.processPipeline({
audience: {
serviceName: "test-service",
componentName: "kafka",
operationType: OperationType.PRODUCER,
operationName: "kafka-producer",
},
data: new TextEncoder().encode(JSON.stringify({ key: "value" })),
});
};
Congrats on the launch! Your approach to observability and data incidents resolution truely stands out. Keep up the good work!
Congrats on the launch, team! What makes it stand out from other data observability tools?
