DeepEval4Claude

Free eval for your AI agent. No keys, no SaaS

Productivity

Developer Tools

Artificial Intelligence

GitHub

Hi PH — Evgeny here. Quick story behind this one. I'm a solo builder. I ship Claude-powered tools without a QA team, without a peer reviewer, and usually without enough sleep. For a long time my "eval pipeline" was: read the agent's output, squint at it, ship it, hope. A few months ago I noticed two things kept biting me: The agent was too agreeable. I'd give it a half-baked premise. It would build a beautiful answer on top of that premise without ever pushing back. I'd ship the answer and only later realize the premise was wrong. The agent quietly chose interpretations. I'd give it an ambiguous input. Instead of asking, it would pick one path and run with it. The output looked great. It was solving the wrong problem. I tried the usual eval frameworks — Ragas, G-Eval, LangSmith. They're good at "did the model hallucinate?" They're not built to catch "did the model suck up to me?" or "did the model silently make a judgment call?" Those are different failure modes. So I built deepeval-bcg for myself, and now I'm putting it out there. What it does You run /deepeval-run path/to/your-agent-output.md inside Claude Code. It scores your output on 8 dimensions using the actual rubric BCG uses to evaluate analysts (calibrated 1–3 scale, weighted). Then it runs a small Skeptic Agent that throws three adversarial probes at the output — an ambiguity probe, a sycophancy probe, and a steelman-the-opposite probe. Then a Novelty Stack that asks "if I search-replace the company name, does this analysis still read the same?" If yes, the output is generic boilerplate dressed up in nice formatting. You get back a PASS / REVISE / FAIL verdict, per-dimension scores, and a concrete fix directive. Whole thing runs in about 30 seconds. Who it's for Honestly: people like me. Solo founders shipping a Claude-powered product without a team to peer-review the outputs. Indie AI engineers building agents as side projects or small SaaS tools, where "looks fine to me" is the only QA step. Claude Code power users who want a sanity check on their agent's work before it goes to a user. If your agent produces strategic memos, research summaries, briefings, plans, recommendations, or anything where the quality of the thinking matters more than the correctness of facts, this skill is built for you. What I learned along the way (the things that surprised me) Sycophancy is invisible to the model that produced it. A model judging its own output for sycophancy is like asking someone to spot their own accent. The Skeptic Agent works because it's a structurally separate prompt with adversarial framing baked in. Subtle but it matters. A verbatim rubric beats a paraphrased one. Early on I paraphrased the BCG anchors into "cleaner" language. Scores drifted week to week as Claude reinterpreted my paraphrases. I went back to the verbatim original anchors and drift dropped noticeably. Lesson: don't get clever with your eval criteria. No-API-keys was a constraint that turned into a feature. I started this because I didn't want to pay another vendor for evals on a side project. It turns out a lot of solo builders feel the same way. The skill runs entirely inside your existing Claude Code session — no second bill, no key juggling, no signup wall. What it isn't Not a tracing / observability platform. If you need production tracing, pair it with LangSmith or similar. Not a faithfulness/RAG eval. Ragas is great at that — this complements rather than replaces. Not a closed-source SaaS. MIT licensed, no telemetry, no upsell on the core. If it's useful, star the repo; if it's not, fork it and rip out what you don't need. How I hope you'll use it Install it. Run it on whatever your agent shipped this week. If it surfaces something useful, tell me in the comments. If it gets something wrong, especially tell me — there's a built-in feedback button that fires off a pre-filled GitHub issue with one click, and I read every one. /plugin marketplace add EvXata/deepeval-bcg /plugin install deepeval@deepeval-bcg /deepeval-amazon That last command runs a complete eval on a bundled real example so you can see the output before pointing it at your own work. Thanks for reading. Genuinely curious what issues other developers are seeing in their agents — drop them below.