PinchBench

Find the best AI model for your OpenClaw

Open Source

Developer Tools

GitHub

Oh wow, the timing is amazing. I installed OpenClaw for the first time yesterday and was genuinely confused about which model to choose. I ended up using an OpenRouter API key with auto model selection, but the model choices felt a bit random. I’m really glad this product launched today, I’ll definitely be using this benchmark.👏

With PinchBench testing real world tasks instead of synthetic benchmarks, how do you decide which tasks go into the benchmark suite and how often do you rotate them to avoid overfitting? Congrats on the launch!

How do you make sure the results from PinchBench reflect real-world use especially when different projects have different complexity, tools and edge cases?

This is exactly what I was looking for. However, tasks should be scoped and agents should be ranked depending on task category.

Imho the most important agent to determine is the main one, the orchestrator, the one you talk to. But then, you will eventually want different subagents specialized in different tasks (and ideally not as expensive, depending on the task at hand). For those, the "best" agent (in terms of value for money) could be something else (i.e., for a simple but broad internet search, gemini flash is often more than enough).

Okay, this is genuinely useful. I've been picking models for coding tasks based on whatever benchmark thread showed up in my feed that week, which is a terrible way to make that decision.

The cost dimension is what gets me. Success rate matters, but if a model takes 3x longer and costs 4x more to get there, that changes the math completely, depending on what you're building. Glad someone's actually measuring all three together.

Curious how you're defining task success — is it automated test output or is there a human eval component? That part always feels like the hardest thing to get right in coding benchmarks.

Congrats on shipping. The 🦀 was not lost on me.

Not just Jensen - y'all gotta know which model's best for your claws!

And y'all can contribute to it, because it's open source 🫶
Great job @realolearycrew !!

Nice benchmarks at the end of the use cases! I would like to see more benchmarks relocated to different levels of tasks (no coding).

Benchmarks like SWE-bench (and agent eval harnesses built around it) are the default reference point for coding agents—what does PinchBench capture about *OpenClaw-in-the-loop* behavior (tool selection, memory, retries, file ops) that SWE-bench-style evaluations systematically miss, and where do you think SWE-bench is still the better signal?