Prefactor

Evaluate your AI Agents in real-time

SaaS

Developer Tools

Artificial Intelligence

Visit WebsiteSee on Product HuntTwitter ⧉LinkedIn ⧉

The evals pass then production drifts problem has a close cousin in ordinary end to end testing, where a suite goes green on a dev laptop and red on a slower CI box. Same root cause both times: the check was written against conditions that only held in one environment.

What I would want to know here is how you handle the scoring being non deterministic itself. If the judge is a model, its verdict drifts too, so a regression alert could mean the agent changed or the scorer changed. Do you pin the scorer version and replay old runs through new scorers to separate those two?

Congrats on the launch.

This is such a valuable tool. As someone who ships agents to clients, an automated testing/scoring tool would really scale my QA processes.

Congrats on the launch! How do you evaluate the AI Agents is it deterministic or using LLMs by providing the truth table?

Congrats on launching! The "did it actually do it, or just say it did" framing will ring true for anyone who has put an agent in front of real users. The 25k spans a month free tier makes it easy for a small team to give this a proper try. How do the custom spans that pull context from Linear or GitHub work in practice, do you write those checks yourself or pick from templates?

Matt, following on from the forum thread where I said the zero-token layer does most of the work: the thing I keep bumping into is that every deterministic check I own exists because something already went wrong once. Banned phrases, schema conformance, the contradicts-its-own-inputs assertion, all of it got written the morning after a bad output shipped. So my free layer's coverage is really a map of my incident history, and the failure it structurally cannot see is the one I haven't met yet.

That interacts awkwardly with scoring 100% of traffic. Full coverage on rules-someone-wrote reads as all-green on the dashboard, and all-green on 100% is a much stronger-sounding claim than all-green on a 30% sample, which is the version everyone already knows to distrust.

You told Yolanda the pattern layer asks how the agent's behaviour differs "from what you know to be the case", and that phrase is where my question lands, because that's still a declared baseline. Is there anything that flags a run as unusual without someone having said what usual is, off span shape, tool sequence, turn count? That's the piece that catches the first instance rather than the second, and it's the only part of this I've never managed to build for myself.

Congrats Matt, Simon and team, this thread is a masterclass! One angle nobody’s asked: your pitch says built for engineering teams, but at our company the agents are built and owned by ops and marketing people like me, not engineers. Once the SDK is wired up, can the business owner of an agent actually live in Prefactor day to day, reading quality scores and approving held runs without understanding what a span is? The 40-agents-no-honest-answer problem exists at non-engineering companies too, we just have nobody to hand the dashboard to.

congrats on the launch! curious as to what metrics they evalaute the other AI agents on

Real-time eval is the piece most agent stacks skip until something breaks in prod. Curious how you handle scoring when there's no ground-truth label — is it rubric-based, or are you comparing against a reference trace? Congrats on the launch.

the "too many tools / looping too frequently" answer is close but not quite the failure mode I'd worry about most. we do voice agents at Dial, and the thing that never shows up in a transcript-based eval is time-to-respond. an agent can land a perfectly correct final answer and still lose the interaction because it took 4 seconds of dead air to get there and the caller hung up or talked over it. does Prefactor score per-turn latency as its own quality dimension, or is timing something teams have to instrument themselves as a custom span?

we're all out here shipping agents with our eyes closed and calling it deployment 😅 congrats on a great launch @ethan_lee8 @matt_doughty

The product looks great! What agents I can try it on? Does new Kimi K3 supported?

the re-read-and-compare pattern people are describing here (rereading the page, querying the row) works because the outcome leaves a durable artifact you can go check after the fact. what about agents whose whole job is a live voice call - once the call ends there's no row to reread, the only ground truth is whatever got captured live during the call itself. curious if Prefactor's eval model assumes there's always some external state to verify against, or if it can score purely off the conversation transcript/audio when that's genuinely the only signal that ever existed.

"Most agents pass their evals and fail in production" is almost word for word what we say about backtests — they look great in-sample and melt the moment they trade live. The failure mode I keep running into is that the eval set slowly becomes the thing you tune against, so the score stops measuring generalization and starts measuring fit.

Does Prefactor do anything to preserve a holdout — scoring against runs the team hasn't looked at, or splitting eval traffic by time rather than at random? Real-time scoring on every run sounds exactly right for catching drift; I'm just curious how you keep the scorer from becoming the target.

Congrats on the launch.
When teams try Prefactor for the first time, how do teams catch the first bad run before users notice?

kind of worried this just becomes the new green checkmark people stop questioning. same problem, one layer up?

The "tokenless until you actually need an LLM" design is the part that stands out, most eval tools default to LLM-as-judge for everything and eat the cost whether it's warranted or not. Given you're scoring 100% of production traffic, curious how Prefactor handles agents that call other agents (sub-agent chains). Does a breach in a nested sub-agent bubble up and pause the parent run too, or does each agent in the chain get evaluated as its own isolated instance?

That's exactly the gap that matters. An agent can produce a perfect transcript and still fail the task in the real world. We've seen jobs report "success" while nothing actually changed. That's why the real evaluation isn't what the agent said—it's what actually happened. Can Prefactor validate external outcomes, like confirming a page was updated, a record was written, or an API state changed, instead of only grading the execution log?

I really like the idea of tracking how the agent is working, as it is performing actions. Being more proactive is better than finding out something went wrong several days ago. It's not to be overlooked that the interruption isn't just killing the agent run, but handing it to a human too. I am going to have to play around with this and my own agents.