GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down.
Hi Product Hunt,
Today, we are excited to be open-sourcing GPTRouter, an LLMOps tool we have been using internally at Writesonic for handling millions of monthly requests for our users.
✅ Universal API for 30+ LLMs, Vision and Image Models
✅ Smart Fallbacks based on latency and uptime
✅ Automatic Retries
✅ Supports streaming
Since embracing OpenAI GPT-3 in production in 2020, we at Writesonic have been serving millions of users and faced the typical scaling pains with generative AI models:
1. Dependency on a single model risked total downtime.
2. Latency issues with models like GPT-4 affected user experience.
3. Integrating various models was tough due to different APIs and SDKs.
🌟 Early this year at Writesonic, we set out with a clear vision: to become model agnostic.
Faced with single-model limitations and diverse AI challenges, we began building GPTRouter - our bespoke solution to navigate and thrive in a multi-model AI world.
🔗 With GPTRouter's Universal API, you're the master of AI models.
Swap between OpenAI, Azure, Anthropic, Replicate, Cohere & more with just one line of code.
It simplifies model management to a great extent.
🛡️ Downtime isn't an option.
GPTRouter's Smart Fallbacks mean your service is always on.
You can define a hierarchy of models for each use case. GPTRouter will constantly check for uptime/downtime, latency and other factors, and automatically fallback to the next best model with zero interruption.
♻️ Say goodbye to manual retries.
GPTRouter does the heavy lifting with Automatic Retries for failed requests, keeping your AI services sharp and consistent.
🌟 GPTRouter's Edge:
✅ Universal API for seamless model switching.
✅ Smart, automatic fallbacks for continuous service.
✅ Reduced latencies for quick interactions.
This is just the starting point. We are also working on integrations with LangChain and LlamaIndex.
Additionally, we will also be open sourcing our frontend LLMOps layer that provides a playground to test multiple models in parallel, keep a tab on the latencies for each model, track tokens and costs for each model and user all in one place.
📈 With GPTRouter, we're not just solving our problems at Writesonic; we're offering a solution to the startups and companies looking to make use of Generative AI in production.
We are looking forward to seeing how developers leverage GPTRouter in their own use cases.
Thank you!
This looks like a great solution for avoiding any downtimes with OpenAI models. I'm curious to know how the smart fallbacks are determined based on uptime and latency. Also, is there any way to customize the automatic retries feature? Keep up the good work!
Thanks for creating this incredibly innovative tool. I can immediately see several great use cases and I look forward to beginning to play around with it.
Wow, this product is a must-have for LLM work! With automatic retries and streaming, it boosts the already impressive gateway. Its ability to stay operational during OpenAI downtimes shows how reliable and robust it is.
Hats off @samanyou_garg for this awesome product!
Congratulations on open-sourcing GPTRouter!
The challenges you've addressed, from model dependencies to latency issues, resonate with many in the generative AI space. 🚀
The idea of being model-agnostic with GPTRouter's Universal API is a game-changer!
The ability to seamlessly switch between various LLMs, Vision, and Image Models with just one line of code simplifies the entire model management process.
The Smart Fallbacks and Automatic Retries features are crucial for ensuring continuous service and consistent AI performance. ⚙️ It's impressive to see how GPTRouter addresses real-world issues and enhances the reliability of AI services.
Looking forward to seeing how developers leverage GPTRouter in their projects.
Keep up the fantastic work and innovation!
Does this handle anything with rate limiting, or does it allow for multiple instances of the same kind of model? we have azure openai deployments in multiple regions for instance.
Smart fallbacks, automatic retries, streaming? GPT Router is the LLM dream team conductor!
Great work Samanayou! You've made an amazing way to help us stay connected and avoid downtimes with one unique API. Kudos to you for creating this open source powerhouse!
This is great. It will save everyone the time to write same wrappers and error handling. Kudos to the team.
I think this is amazing since it's open source and customizable so people can use it as a base and build new products or modify it as per their needs instead of the modification living behind an enterprise paywall it's out in wild that makes the MOAT for this
