memtrust

The benchmark vendors don't run on themselves

Open Source

Developer Tools

Artificial Intelligence

GitHub

Visit WebsiteSee on Product HuntNpm ⧉Github ⧉

The catalyst for building this was MemPalace. Their README originally touted a 100% LongMemEval figure (measured with Haiku reranking), which turned out to be irreproducible from the repo's own benchmark scripts and was eventually pulled. A second, still-cited 96.6% figure is largely the result of ChromaDB's default embeddings doing the heavy lifting in raw mode, rather than MemPalace's actual architecture. Furthermore, their claim of "lossless" AAAK compression drops that 96.6% down to 84.2% in practice, a 12.4-point gap from what "lossless" implies. This was entirely documented by MemPalace's own users in [Issue #27] (opened April 7). Two PRs attempting to fix the reporting (#433 and #729) were closed unmerged on April 12, with #729 closed just seven minutes after it was opened. We decided it was time to build a tool that checks vendor claims independently, using the exact same methodology across the board. The reason we are confident enough to publish this benchmark is that it caught a fictional API in its own code before we ever pointed it at a third-party vendor. Every earlier version of our MemPalaceAdapter called a mempalace.Palace(storage_path=...) class with .remember(), .recall(), and .invalidate() methods. That class does not exist in the real installed package (running python3 -c "import mempalace; hasattr(mempalace, 'Palace')" returns False). Every "passing" test we had was exercising a hand-written fake. The tool caught this, and we've since rewritten the adapter against the real mempalace.mcp_server functions. Right now, only one backend has a full run: mem0_direct. This is the self-hosted mem0ai OSS library running in-process through memtrust's adapter, backed by local Qdrant + OpenAI and not Mem0's hosted Platform API. Here is what we found on a small sample size: - Contradiction Detection: 7 of 7 cases were silently overwritten. (100% silent-overwrite rate, 0% flagged as a conflict). - Compression Fidelity: Across 5 cases, mean literal fidelity was 31.6%. - Extraction Quality: Across 15 cases, all 12 deliberately-junk entries were correctly rejected, but all 3 deliberately-valid entries were lost on retrieval. Getting this run working also exposed a bug in mem0ai itself. Its default model string (gpt-5-mini) isn't recognized by mem0's own reasoning-model detection, which checks for gpt-5o-mini. Anyone setting OPENAI_API_KEY gets a 400 on every LLM-based extraction call right out of the box. Our adapter now works around this. Mem0, MemPalace, Zep, and OpenViking are all real, useful projects with real communities. This isn't an attack on any of them. It's an attempt to give teams a verifiable number. Zep/Graphiti and OpenViking aren't live-benchmarked yet, but they're next on our list. If you are evaluating memory systems, drop a comment with which backend you are eyeing and what you care most about (contradiction handling, compression fidelity, retrieval at scale), and we'll share what we've found so far. If you have live credentials and want to help produce the first independently-run MemPalace or Zep benchmark, please open an issue!