This product was not featured by Product Hunt yet. It will not be visible on their landing page and won't be ranked (cannot win product of the day regardless of upvotes).
Statistical regression testing for LLM agents. Run versions A and B 50 times to get a p-value, Cohen's d, and a 95% CI proving whether behavior actually shifted. While DeepEval, Braintrust, and Promptfoo test single responses against a threshold, none measure distribution drift. It is completely self-hostable under an Apache 2.0 license and requires no SaaS subscriptions. Works natively with LangGraph, OpenAI Agents SDK, CrewAI, and LangChain LCEL. Start testing: `pip install agent-regress-cli`.
The recurring failure mode is frustratingly common: an agent is updated, standard evals stay green, but performance noticeably degrades a week later. Two real-world examples catalyzed the development of this tool:
- LangGraph PR #4486: Added node-level result caching, which quietly masked the repeated-sampling variance a regression test depended on, something a plain threshold check has no way to catch.
- CrewAI PR #6134: Fixed a bug where file tools leaked absolute filesystem paths in responses. A standard quality scorer never flags this because it only checks if the answer is correct, not what is hidden in the payload.
These examples are part of a larger validation pass encompassing 29 PRs and 239 rows across LangGraph, CrewAI, and the OpenAI Agents SDK. The full data breakdown is available in the pr-analysis.md documentation on github.
Instead of grading a single response, agent-eval statistically compares two whole batches of runs. By running versions A and B 50 times each, the tool executes:
- A Mann-Whitney U test (requiring no assumption that scores are normally distributed).
- A bootstrap confidence interval to confirm if the shift is real.
- Cohen's d to determine if the shift is large enough to actually matter.
A p-value under 0.05 by itself is not enough to call it a regression. The effect size is required for definitive proof.
As part of statistical guardrails, two limits are utilized to ensure data integrity.
- Under 30 runs per version: Triggers a warning due to a drop in statistical power.
- Under 10 runs: Returns INSUFFICIENT_DATA rather than guessing at an answer.
This is one piece of a larger observability, testing, and benchmarking infrastructure designed for production agents. We are already planning for the next version.
Should AutoGen or the Vercel AI SDK be prioritized next? What do you suggest?
No comment highlights available yet. Please check back later!
About Agent-Eval on Product Hunt
“Statistical regression testing for LLM agents”
Agent-Eval was submitted on Product Hunt and earned 0 upvotes and 1 comments, placing #124 on the daily leaderboard. Statistical regression testing for LLM agents. Run versions A and B 50 times to get a p-value, Cohen's d, and a 95% CI proving whether behavior actually shifted. While DeepEval, Braintrust, and Promptfoo test single responses against a threshold, none measure distribution drift. It is completely self-hostable under an Apache 2.0 license and requires no SaaS subscriptions. Works natively with LangGraph, OpenAI Agents SDK, CrewAI, and LangChain LCEL. Start testing: `pip install agent-regress-cli`.
Agent-Eval was featured in Open Source (68.7k followers), Developer Tools (517.4k followers), Artificial Intelligence (475.7k followers) and GitHub (41.4k followers) on Product Hunt. Together, these topics include over 233.2k products, making this a competitive space to launch in.
Who hunted Agent-Eval?
Agent-Eval was hunted by Sourav Nandy. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Want to see how Agent-Eval stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
Sourav Nandy