Meta is releasing three models: The new 3.1-405B and upgrades to their smaller models: 3.1-70B and 3.1-8B. If 405B is as good as the benchmarks indicate, this would be the first time an open source model rivaled the best closed models—a profound shift.
This could be The One - the open source model that closes the gap with the top closed models like GPT-4o / Claude-3.5. It's a "curves-crossing" moment reminiscent of how the Intel vs. ARM approaches played out, and perhaps with similar profound effects on the landscape.
If you're in SF, join us tonight for a meetup and 405B panel including founders of Vercel and JuliusAI: https://lu.ma/4es9bfgs
Also, one-time only: Launch your product TODAY using Llama-3 405b and we'll feature it (this won't prevent you from launching in the near future). Plus the top launches will be eligible to demo tonight after the panel.
In private conversations with launch partners, Meta has emphasized 405B's reasoning capability and multilingual abilities. This would seem to have big implications for interfaces, especially voice interfaces. Are people finding the model lives up to this, practically?
Som more thoughts from a friend, @kwindla (Daily.co), who is a launch partner for 405B:
"1. 405B beats GPT-4o on 11 of 13 widely used benchmarks. And Meta/Fair has a history of being careful about these benchmarks, so they almost certainly went to a lot of effort to not let training data leak into test, etc. No open source model has previously come close to GPT-4o/Claude-3.5. It’s a huge, huge deal if this is accurate and reflects the quality of “reasoning” the model can do.
2. The two smaller 3.1 models (70B and 8B) also made big leaps in benchmark performance. That indicates that Meta’s strategy for training/distilling is working. Having models that are small enough to run on single devices (or, on LPUs, very very very fast and inexpensively) that are this good may be equivalent to leap-frogging GPT-4o-mini. This also gives people the opportunity to experiment with fine-tuning really good models and with doing architecture/merge experiments.
3. Big models have a different “tone/vibe” than small models. 3-70B was a pretty good model in a lot of ways, but as a conversational agent it just didn’t feel as good qualitatively as GPT-4o and Claude-3.5. That feel really matters in things like consumer-facing voice chat use cases. If 405B is approximately as good as the proprietary models on benchmarks, *and* matches their “vibe” for the first time, that’s truly exciting for a whole range of next-generation conversational/interactive use cases."
This is something serious that can be a game-changer. I am surprised at the votes LOL - maybe the voting mechanism truly need some changes.
Wait a minute. Llama 3.1 can't be called "open-source."
It looks impressive, and I’m excited to test it. But it doesn't fit the proposed definitions of "open-source AI."
For LLMs to be truly "open-source," they need to include:
1: Detailed info about the training data so that someone can recreate a similar system with similar data.
2: The source code used to train and run the system.
This is an important requirement, not a simple detail.
While "open-source" has clear definitions in traditional software, its definition is still a work in progress for AI.
Concepts are evolving, but we need to be more careful with terms and separate PR buzz from reality.
For more on the definition of Open Source AI and this debate, check out: https://opensource.org/deepdive/...
Great competition is great news for the market, especially if it's open source.
I'm super stoked to try it out, especially self-hosting the smaller models.
Congrats to the team at Meta :)
Good hunt Rajiv! The best of the best AI models will be the ones that are open source.
Meta continued commitment to open models is great for everyone building tools with LLM's.
Llama 3.0 was already an amazing model, and the one I used with Career Tailor AI. An upgrade that's still open is incredibly welcome.
Allowing outputs from the largest 3.1 model to be used in training is going to be a game changer for training smaller more specific models. 🤯
Very eye-catching way for big brands to announce important product launches! Like the smart design!
This is an exceptionally interesting way to drop a SOTA Open Source Model. Kudo's to the ProductHunt team for securing this launch and traffic.
