This product was not featured by Product Hunt yet. It will not be visible on their landing page and won't be ranked (cannot win product of the day regardless of upvotes).
Tracely
Production failures become regression tests for AI agents
Tracely grades every agent trace as it lands, clusters failures into issues, freezes bad runs into hermetic replayable test cases — and blocks the PR that would ship them again. The trace is the test: no hand-authored datasets. Open source, MIT.
I'm Julien, and I built Tracely because of a loop every agent builder knows: your agent breaks in production, you see it on a dashboard, you nod… and then you sit down to hand-write an eval dataset trying to reproduce what production already handed you for free.
Tracely flips that. The trace is the test:
🔍 Observe — one line of setup: tracely.init(instrument="auto") auto-captures your OpenAI / Anthropic / LangChain / Gemini / Mistral calls with zero span code; a manual API (@observe, agent/tool/llm/retriever/guardrail spans) covers your own logic; and raw OTLP works from any language. The payoff is the read: traces render as conversations — turns → steps → a span waterfall with full I/O on every node. I obsessed over making this the most readable trace view in the category.
⚖️ Detect — this is where Tracely goes deepest. Evaluators are columns on the trace table, not a separate tab, and they grade every run as it lands:
Three levels: conversation, message, or step — and step-level judges can target exactly the span types you care about (tool calls only, generations only, chains, thinking).
Typed outputs, not just a score: define the judge's output schema — booleans, numbers, strings, strict enums the judge cannot escape. A verdict like sycophancy_type: excessive_validation · severity: moderate is queryable data, not prose.
Sequential or batch execution: batch judges run independently; sequential judges receive the previous metrics' results injected as context, so you can chain "did the tool succeed?" → "was the answer faithful to what it returned?". Message-level sequential judges also see the conversation's rolling summary (@HISTORY), built in real time as turns land.
Spend control built in: per-evaluator targeting by agent/env plus deterministic sampling, so the same trace always gets the same judges — and advisory evaluators record a verdict without flipping the PASS/FAIL roll-up while you're still tuning them.
🧩 Triage — failures cluster into issues. 31 broken runs = one issue with a count, not 31 rows.
🧊 Test — one click freezes a failing run into a hermetic case: recorded input, recorded tool/LLM outputs as fixtures, and a fail-to-pass contract.
🚫 Ship — tracely gate replays the suite on every PR and blocks the one that reintroduces the failure. Replay is against fixtures, so it costs $0 in model spend. There's also an adversarial mode: a red-team model improvises against your agent's live endpoint in CI.
It's open source (MIT) — 370+ GitHub stars in its first months — self-host everything with one `docker compose up` (`pip install tracely-ai` for the SDK). Don't want to run infra? There's a hosted cloud with a free plan at tracely-studio.xyz: multi-workspace with team invites, and you bring your own OpenRouter API key for the LLM judges — scoped per workspace, never a shared server key (more providers coming). No key configured? Everything still runs: structural checks keep grading and the LLM judges switch off gracefully.
Two things I'd especially love feedback on: the judge-calibration flow (label judge verdicts against human review, catch an over-flagging judge before it gates a release) and the adversarial scenarios.
No comment highlights available yet. Please check back later!
About Tracely on Product Hunt
“Production failures become regression tests for AI agents”
Tracely was submitted on Product Hunt and earned 4 upvotes and 1 comments, placing #113 on the daily leaderboard. Tracely grades every agent trace as it lands, clusters failures into issues, freezes bad runs into hermetic replayable test cases — and blocks the PR that would ship them again. The trace is the test: no hand-authored datasets. Open source, MIT.
Tracely was featured in Open Source (68.7k followers), Developer Tools (517.3k followers), Artificial Intelligence (475.7k followers) and GitHub (41.4k followers) on Product Hunt. Together, these topics include over 233.2k products, making this a competitive space to launch in.
Who hunted Tracely?
Tracely was hunted by Julien Wuthrich. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Want to see how Tracely stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
Julien Wuthrich
Hey Product Hunt 👋
I'm Julien, and I built Tracely because of a loop every agent builder knows: your agent breaks in production, you see it on a dashboard, you nod… and then you sit down to hand-write an eval dataset trying to reproduce what production already handed you for free.
Tracely flips that. The trace is the test:
🔍 Observe — one line of setup: tracely.init(instrument="auto") auto-captures your OpenAI / Anthropic / LangChain / Gemini / Mistral calls with zero span code; a manual API (@observe, agent/tool/llm/retriever/guardrail spans) covers your own logic; and raw OTLP works from any language. The payoff is the read: traces render as conversations — turns → steps → a span waterfall with full I/O on every node. I obsessed over making this the most readable trace view in the category.
⚖️ Detect — this is where Tracely goes deepest. Evaluators are columns on the trace table, not a separate tab, and they grade every run as it lands:
Three levels: conversation, message, or step — and step-level judges can target exactly the span types you care about (tool calls only, generations only, chains, thinking).
Typed outputs, not just a score: define the judge's output schema — booleans, numbers, strings, strict enums the judge cannot escape. A verdict like sycophancy_type: excessive_validation · severity: moderate is queryable data, not prose.
Sequential or batch execution: batch judges run independently; sequential judges receive the previous metrics' results injected as context, so you can chain "did the tool succeed?" → "was the answer faithful to what it returned?". Message-level sequential judges also see the conversation's rolling summary (@HISTORY), built in real time as turns land.
Spend control built in: per-evaluator targeting by agent/env plus deterministic sampling, so the same trace always gets the same judges — and advisory evaluators record a verdict without flipping the PASS/FAIL roll-up while you're still tuning them.
🧩 Triage — failures cluster into issues. 31 broken runs = one issue with a count, not 31 rows.
🧊 Test — one click freezes a failing run into a hermetic case: recorded input, recorded tool/LLM outputs as fixtures, and a fail-to-pass contract.
🚫 Ship — tracely gate replays the suite on every PR and blocks the one that reintroduces the failure. Replay is against fixtures, so it costs $0 in model spend. There's also an adversarial mode: a red-team model improvises against your agent's live endpoint in CI.
It's open source (MIT) — 370+ GitHub stars in its first months — self-host everything with one `docker compose up` (`pip install tracely-ai` for the SDK). Don't want to run infra? There's a hosted cloud with a free plan at tracely-studio.xyz: multi-workspace with team invites, and you bring your own OpenRouter API key for the LLM judges — scoped per workspace, never a shared server key (more providers coming). No key configured? Everything still runs: structural checks keep grading and the LLM judges switch off gracefully.
Two things I'd especially love feedback on: the judge-calibration flow (label judge verdicts against human review, catch an over-flagging judge before it gates a release) and the adversarial scenarios.