RunInfra

Describe the AI model you need and get an optimized AI

API

Developer Tools

Artificial Intelligence

Visit WebsiteSee on Product Hunt

Generating custom CUDA kernels automatically via the Forge agent is a step further than most optimize-your-model tools that stop at quantization. For less common architectures where kernel patterns aren't well-trodden, does it fall back to a safer generic path, or is manual tuning still needed there?

The scale-to-zero + pay-per-million-tokens combo is the part I'd test first. I’ve had small agent prototypes where idle GPU cost felt silly. Curious how you decide when to generate custom CUDA vs just quantize/route to an existing runtime?

Abstracting model selection and kernel tuning behind a plain description is a good bet for teams without an ML infra person. How opinionated is it, does it pick the architecture and hardware or mostly optimize what you hand it? The gap between 'I need X' and a deployed model is where most people get stuck.

how does the pricing actually work when you hit something like a custom CUDA kernel being generated, is that a flat fee or does it burn through tokens while forge is reasoning?

This looks cool! I have started playing around with inference optimisations! Wondering is there a way to learn and contribute at the same thime.

Spent a weekend chatting with RunInfra to spin up a voice-to-text pipeline and the custom CUDA kernel step actually beat the latency i was getting on my old setup. Pricing by the million tokens with scale to zero is a nice fit for the random bursts of traffic i get from indie clients.

how does the cuda kernel generation actually work in practice, does forge just spit out a kernel you can drop into vllm or does it need a custom serving stack on your end

how does the per-token pricing actually compare to something like runpod or modal when running something like a 70b quantized model for a few hours a day?

how does the pricing per million tokens actually compare to something like runpod or modal when you're running a custom kernel workload, especially at lower utilization?

Tried it with a small vision model this morning and the speed jump over my usual setup was noticeable right away, plus the per-token pricing is way easier to stomach than the GPU bills I was getting before.

Tried it with a vision pipeline and the custom CUDA kernels actually beat the hosted version I was using. The plain-language setup is refreshing, no YAML rabbit holes.

How does the custom CUDA kernel generation actually work in practice, does Forge learn from existing kernels or write them from scratch, and what happens if the generated kernel underperforms the standard one at runtime?

Spent a few minutes describing a doc search use case and the generated API was already hitting it faster than my usual setup, the per-token pricing is a nice touch too. Curious how the custom CUDA kernels hold up on weirder workloads.

Tried spinning up a vision model just by describing it and it actually returned a working endpoint, no dashboard digging required. The custom CUDA kernel generation is a wild flex for a chat interface.

How does the Forge agent actually decide when to write a custom CUDA kernel versus just relying on quantization, and does that choice change the price I pay per million tokens?

Tried spinning up a custom voice pipeline in the chat and it actually worked without me touching a config file. The CUDA kernel generation for a smaller Llama variant was way faster than I expected, ran cooler on my GPU too.

Auto-generating custom CUDA kernels is the part that would make me nervous to trust blindly. A kernel can be fast and still be subtly wrong on edge cases, like a numerically unstable softmax or a padding bug that only shows up on odd sequence lengths. What's the testing story before a generated kernel goes into a production API, do you diff outputs against the reference implementation across a range of inputs first?

Osama, the part that lands for me is not having to become an infrastructure expert just to get a model running properly. That barrier has quietly killed plenty of good ideas, so seeing it lowered is refreshing.