This product was not featured by Product Hunt yet. It will not be visible on their landing page and won't be ranked (cannot win product of the day regardless of upvotes).
One binary. No dependencies. Drop it in and run. Up to 4.62x faster prefill than llama.cpp and 5.53x faster batched decoding than vLLM. jitllm detects your hardware and optimizes kernels and model placement across CPUs and GPUs. Run open-weight flagships, even larger than memory. OpenAI- and Anthropic-compatible APIs, desktop and terminal apps. Linux, macOS and Windows.
Inference engines are bulky and rigid. There’s a lot to set up, and even after you get a model running, getting the most out of your hardware is another job.
I wanted to build something easier to live with: one binary you can drop onto a machine and start running models. Something that does more of the work of figuring out the hardware itself.
The inspiration came from operating systems and JIT compilers, two things I've worked on a lot. An OS detects hardware, uses the right drivers, schedules processes, and manages memory. A JIT compiler generates code for the machine it's running on. jitllm brings those ideas together to make inference faster, easier to run, and more flexible.
That's why I call it an operating system for LLM inference. It detects your CPUs and GPUs, generates optimized kernels, and orchestrates model placement at runtime. Models larger than memory page from disk. Layers can move between devices while keeping their conversation state, so changing placement doesn't mean starting over.
jitllm is easy to set up, deploy, and use, and it's mighty fast! You can find the benchmarks and get started at https://jitllm.org
jitllm was submitted on Product Hunt and earned 4 upvotes and 3 comments, placing #44 on the daily leaderboard. One binary. No dependencies. Drop it in and run. Up to 4.62x faster prefill than llama.cpp and 5.53x faster batched decoding than vLLM. jitllm detects your hardware and optimizes kernels and model placement across CPUs and GPUs. Run open-weight flagships, even larger than memory. OpenAI- and Anthropic-compatible APIs, desktop and terminal apps. Linux, macOS and Windows.
jitllm was featured in Developer Tools (521.5k followers), Artificial Intelligence (481.6k followers), GitHub (41.4k followers) and Tech (635k followers) on Product Hunt. Together, these topics include over 422.7k products, making this a competitive space to launch in.
Who hunted jitllm?
jitllm was hunted by Samy Fodil. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Want to see how jitllm stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
Samy Fodil
Hey Product Hunters,
Inference engines are bulky and rigid. There’s a lot to set up, and even after you get a model running, getting the most out of your hardware is another job.
I wanted to build something easier to live with: one binary you can drop onto a machine and start running models. Something that does more of the work of figuring out the hardware itself.
The inspiration came from operating systems and JIT compilers, two things I've worked on a lot. An OS detects hardware, uses the right drivers, schedules processes, and manages memory. A JIT compiler generates code for the machine it's running on. jitllm brings those ideas together to make inference faster, easier to run, and more flexible.
That's why I call it an operating system for LLM inference. It detects your CPUs and GPUs, generates optimized kernels, and orchestrates model placement at runtime. Models larger than memory page from disk. Layers can move between devices while keeping their conversation state, so changing placement doesn't mean starting over.
jitllm is easy to set up, deploy, and use, and it's mighty fast! You can find the benchmarks and get started at https://jitllm.org