Miqu 70b and Mistral 70b have been finally unveiled with the recent Mistral Medium leak shedding light on highly awaited open-source AI models and their capabilities are showcased.
The AI world is abuzz with the “Miqu-1-70b” model and its potential connection to Mistral AI’s secretive Mistral Medium. On January 28th, “Miqu Dev” mysteriously uploaded “miqu-1-70b” files on Hugging Face, followed by an appearance on 4chan and X.
The model impressed, even rivaling GPT-4 in some areas. Not convinced yet? See N8 Programs‘ post on X below.
Whatever Miqu is, it has some sort of special sauce. It gets an 83.5 on EQ-Bench (evaluated locally), surpassing *every other LLM in the world except GPT-4*. EQ-Bench has a 0.97 correlation w/ MMLU, and a 0.94 correlation w/ Arena Elo. It *beats* Mistral Medium – at Q4_K_M. I… pic.twitter.com/0gOOPjxjPD
— N8 Programs (@N8Programs) January 30, 2024
Mistral Medium leak unveils Miqu 70b and Mistral 70b
On January 28, a user by the name of “Miqu Dev” dropped a bombshell on HuggingFace, a popular platform for AI enthusiasts. They uploaded a set of files that make up the Miqu 70B model, which has been creating quite a buzz in the AI world. This model is being hailed as a potential game-changer, with some claiming that it could give GPT-4 a run for its money.
The model was tested using four professional German data protection training exams, which reflected the actual certification tests required for employees. The results were impressive, with Miqu 1 70B correctly answering 17 out of 18 multiple-choice questions, showcasing its robust understanding of the content. However, it failed to adhere to the instruction of responding with “OK” to acknowledge information, which marks a shortfall in instruction compliance.
When compared to other models, Miqu 70B showed proficient language skills and bilingual abilities. However, it didn’t outperform the Mixtral-8x7B-Instruct-v0.1 model or other high-ranking models like GPT-4, Goliath-120B-GGUF, and Tess-XL-v1.0-GGUF, all of which achieved perfect scores in both testing rounds and adhered to the “OK” instruction.
Is Miqu 70B really a Mistral Medium leak?
The origins of Miqu-1-70B remain a mystery, with some speculating that it’s a leaked version of Mistral-Medium or an older experimental version. The model’s performance has sparked debates and discussions in the tech community, with some claiming that it’s the real deal, while others are skeptical.
Arthur Mensch, the big boss over at Mistral, has finally addressed the rumors. He confirmed that an eager beaver from their early access crowd got a bit carried away and leaked an old model. However, Mensch also hinted that they’ve been cooking up something even better since then, a model that could potentially outshine GPT-4.
An over-enthusiastic employee of one of our early access customers leaked a quantised (and watermarked) version of an old model we trained and distributed quite openly.
To quickly start working with a few selected customers, we retrained this model from Llama 2 the minute we got…
— Arthur Mensch (@arthurmensch) January 31, 2024
If Mistral rolls out an open-source model that’s on par with GPT-4, it could send shockwaves through the AI scene. OpenAI might have to watch its back, as there’s a new contender in town, ready to challenge the status quo. This leak could very well be the spark that lights up a whole new era in AI, where the big names
So is Mistral better than ChatGPT?
Determining whether Mistral is “better” than ChatGPT is difficult because it depends on what you’re looking for in an LLM and how you define “better.” Here’s a breakdown of their strengths and weaknesses to help you decide which one might be a better fit for your needs:
Mistral
Strengths:
- Open-source: Mistral offers several models, including some open-source ones, making them more accessible for personal and research purposes
- Efficiency: Mistral models like Mixtral-8x7B are known for their efficiency, requiring less computational power than comparable models like GPT-4
- Transparency: While not all models are open-source, Mistral tends to be more transparent about their development process and research compared to OpenAI’s closed-source approach with ChatGPT
Weaknesses:
- Performance: Overall, Mistral models tend to score slightly lower than top models like GPT-4 on benchmarks for tasks like text generation and translation
- Accessibility: Some of Mistral’s most powerful models (like Mistral Medium) are not publicly available or require waitlists and access fees
- Limited community: Compared to ChatGPT, Mistral has a smaller user base and community, which could limit resources and support available
ChatGPT
Strengths:
- Performance: GPT-4 currently demonstrates top performance in many benchmarks, particularly in areas like creative text generation and complex dialogue
- Accessibility: OpenAI offers free access to ChatGPT through beta programs and APIs, making it more readily available for personal use
- Large community: The large user base and community create extensive resources, guides, and applications for using ChatGPT
Weaknesses:
- Closed-source: The closed-source nature of ChatGPT raises concerns about transparency and potential biases in the model
- Computational cost: GPT-4 requires significant computational power, making it less accessible for individuals with limited resources
- Ethical concerns: OpenAI’s control over access and limited transparency raise concerns about responsible development and potential misuse of the technology
Ultimately, the “better” option depends on your priorities. If you need an open-source model with good efficiency and transparency, Mistral might be a good fit. If you prioritize top performance and broader community support, ChatGPT might be the better choice.
Featured image credit: Mistral AI.
