TokenProxy

Stop LLM overspending with a high-speed Rust gateway 🦀

SaaS

Developer Tools

Artificial Intelligence

Visit WebsiteSee on Product Hunt

Building TokenProxy: Architecture > Everything else! 🛠️🦀

The real challenge with AI proxies isn't forwarding requests—it's managing state and limits at 10k+ RPS without breaking a sweat. Just knocked out two big pieces of the puzzle:

1️⃣ Partition Cleanup Worker: No more database bloat. Postgres partitions now rotate and clean up automatically.

2️⃣ Supercharged User Context: Every limit, option, and key now loads in one swift go.

Combined with my Redis + Bitcode cache service, our overhead is practically non-existent. ⚡

What's next?

• AES-GCM encryption layer for provider keys.

• The "catch-all" handler for seamless API mapping.

Target MVP: May 31st 📅.

#BuildInPublic #Rust #AIProxy #IndieHackers #ProductHunt #Solopreneur

How I scaled our AI model pricing update engine 10x with zero infrastructure cost 🦀🚀

AI moves fast—model prices change by the minute. Our challenge? Building a worker job to get all supported model pricing data that tracks thousands of models, but keeping it running on a tiny $5/month VPS (2GB RAM).

Traditional INSERT statements were killing our performance. Our server was constantly maxing out CPU and memory, and we were falling behind the data stream.

So, I spent today some time to dive into the PostgreSQL Binary Protocol. Instead of using slow, text-based SQL commands, I wrote a custom Rust procedural macro that streams data directly to the database buffer using the COPY FROM STDIN command.

The Results?

✅ 90% lower CPU usage during data ingests.

✅ Minimal Memory Footprint: The 2GB RAM instance doesn’t even sweat anymore.

✅ Seamless 100k+ Row Batches: We can now process huge data dumps instantly.

It will be useful as well when inserting logs to database.

Scaling doesn’t always require a bigger cloud budget. Sometimes, it just requires replacing comfortable abstractions with raw, byte-level efficiency.

Would love to hear from other makers—what’s the most satisfying "over-engineered" optimization you’ve done to keep your burn rate low?

#IndieHackers #BuildInPublic #RustLang #Postgres #Scalability

🛠 The Architecture (Why it's so fast)

Most LLM proxies add significant latency because they are built on interpreted languages or have complex infrastructure requirements. TokenProxy is different. We built it for sub-millisecond overhead:

• Rust Core: Leveraging Axum and Tokio for non-blocking, high-concurrency request handling.

• Postgres Advisory Locks: We ditched Redis for quota management. By using pg_advisory_xact_lock, we ensure atomic, race-condition-free credit updates directly in the database with zero extra network hops.

• Binary COPY Batching: Our logging engine doesn't do row-by-row inserts. It buffers logs and flushes them using the Postgres Binary COPY protocol, triggered by time, count, or size.

• Unified Streaming: A provider-agnostic engine that transforms SSE (Server-Sent Events) from Claude or Groq into a standard OpenAI-compatible stream on the fly.