Inferock Bench

An independent receipt for every LLM API call

Open Source

Developer Tools

Artificial Intelligence

GitHub

Visit WebsiteSee on Product HuntGithub ⧉LinkedIn ⧉Twitter ⧉

Congrats on the launch of Inferock Bench.

Love the idea of having an independent receipt for every LLM API call. Transparent usage, billing, and overpayment tracking can be a huge win for teams working with multiple LLM providers.

Retries are the part I’d put in giant font. ‘$0.04/call’ means nothing if a successful job secretly takes 6 calls. I want cost per accepted outcome.

The failed-calls and retries breakdown is the part I'd use first. One case I keep hitting might not show up there: a tool call that returns 200 with a silently corrupted value. I measured this on Anthropic models, 40 calls, none flagged the value was wrong, so it bills as a clean success and the retry logic never fires. Can the receipt catch a call that looked fine but wasn't? Or is that out of scope by design?

Congrats on the launch @himashwetha_gowda. Good find @fmerian.

Question regarding accuracy, is it possible it might not be able to distinguish a genuinely billable provider failure from valid hidden token usage, such as reasoning, refusal, cache, or tool-call tokens?

Thanks @fmerian for hunting us. Feel free to ask if anybody has any questions about Inferock Bench.

Does anyone know of any tools that can work with web-based logins (claud.ai etc)

The bill dispute question is interesting. have you personally managed to get a provider to credit a charge after showing them one of these per call receipts or is that still something you are testing?

Which API call actually cost me this money? is a question every AI app eventually needs to answer.

changing the baseURL and apiKey instead of rewriting the application is a nice touch. makes this much easier to test on an existing project.

I like that Inferock Bench focuses on evidence rather than simply showing another dashboard. Having a separate record of every API call could make unexpected billing much easier to investigate.

The retry tracking caught my attention. I’ve seen failed requests become surprisingly expensive, so being able to trace each call would be useful to me.

Love that this sits as a local proxy instead of asking me to route traffic through another cloud service. Keeps my API keys and data where they belong.

I've been burned by silent retries inflating my OpenAI bill before. Having an independent receipt for that would've saved me a painful invoice conversation.

I like the positioning here, it's not trying to replace the provider's billing system, just verify it. That's a smarter pitch than "cost optimization," which every tool claims.

This is relevant to a problem I've had for months. I run agents that call out to multiple models depending on the task complexity and every so often the bill jumps in a way I can't explain from usage alone. If this can pinpoint whether that's failed calls, redundant retries or just legitimate scaling, I'd finally have an answer instead of a guess.

What I really want to know is how this handles historical data. Can I feed it a month of past logs and get a retroactive receipt or is it strictly forward-looking from install? I ask because the overspending I'm most curious about already happened and I'd love a way to audit it after the fact.

Tracking retries alongside failures is a smart addition. Those hidden retries can quietly become a big part of the bill.

I like that it works as a local proxy. Keeping billing and usage data on the developer's machine feels like a thoughtful design choice.