Vicuna-13B AI chatbot has been introduced as an open-source chatbot that is trained using fine-tuning LLaMA with user-shared conversations gathered from ShareGPT. This chatbot has undergone a preliminary evaluation using GPT-4 as the benchmark, which shows that it achieves over 90%* quality compared to other popular chatbots like OpenAI ChatGPT and Google Bard. Vicuna-13B also outperforms other models, such as LLaMA and Stanford Alpaca, in more than 90%* of cases.
Despite being almost twice the size of Stanford Alpaca, Vicuna-13B only costs about $300, which is half the cost of Alpaca. This is because the ShareGPT data, from which Vicuna is trained, is openly accessible, whereas Stanford generated its own data via the OpenAI API. Hence, Vicuna only incurs training costs. Similar to the Stanford model, Vicuna-13B is made available for non-commercial purposes only.
According to the team behind Vicuna, the chatbot has demonstrated significantly better-structured and more detailed answers than Alpaca during benchmark question tests after being fine-tuned with ShareGPT data. The team states that the quality of Vicuna-13B AI chatbot’s answers is comparable to those provided by ChatGPT.
GPT-4 can compare the outputs of various chatbots, but the method is considered “non-scientific” by the Vicuna-13B AI chatbot team, and further research is necessary – Image: LmSys
To further evaluate Vicuna-13B AI chatbot’s performance, the team is using GPT-4, OpenAI’s latest model. GPT-4 enables the team to consistently rank and score different chatbots in a detailed manner, thus providing a more comprehensive assessment of Vicuna’s capabilities.
What is Vicuna-13B AI chatbot?
Vicuna is an open-source chatbot that has been fine-tuned using around 70,000 user-shared conversations obtained from ShareGPT.com through public APIs. The Vicuna-13B AI chatbot is based on LLaMA, which was fine-tuned to optimize its performance. To ensure data quality, the research team converted the HTML format back to markdown and removed inappropriate or low-quality samples. Furthermore, they segmented lengthy conversations into smaller parts that would fit the maximum context length of the model.
The research team developed several enhancements to Stanford’s Alpaca training recipe to improve Vicuna’s performance. These enhancements include:
- Memory optimizations: The team expanded the maximum context length from 512 in Alpaca to 2048, which allowed for better understanding of long conversations. However, this resulted in a significant increase in GPU memory requirements. To mitigate this, the team used gradient checkpointing and flash attention to manage the memory pressure.
- Multi-round conversations: The team adjusted the training loss to account for multi-round conversations and calculated the fine-tuning loss only based on the chatbot’s output.
- Cost reduction via Spot Instance: Training with a 40x larger dataset and 4x sequence length posed a considerable cost challenge. To reduce costs, the team used SkyPilot managed spot, which utilized cheaper spot instances with auto-recovery for preemptions and auto zone switch.
The various optimizations made to Vicuna enable the chatbot to better comprehend and respond to complex conversations. Additionally, the cost reduction strategies make Vicuna a more affordable option for developers and researchers building chatbot systems.
To train Vicuna-13B AI chatbot, the research team collected approximately 70,000 conversations from ShareGPT.com, a platform where users can share their ChatGPT conversations. The training scripts provided by Alpaca were modified to better handle long sequences and multi-round conversations. The team used PyTorch FSDP on eight A100 GPUs to train Vicuna in just one day.
The demo implementation employed a lightweight distributed serving system capable of serving multiple models using distributed workers. This system supports flexible GPU worker plug-ins from both cloud-based and on-premise clusters. To reduce serving costs, the team used a fault-tolerant controller and managed spot features in SkyPilot, which leveraged less expensive spot instances from multiple clouds.
How to use Vicuna-13B AI chatbot?
You can try out Vicuna-13B AI chatbot by following these steps:
- Go to lmsys.org
- Choose “vicuna-13b” from the drop-down menu
- Below you will see a box saying “Enter text and press ENTER”
- Ask whatever you want and click on upload.
That’s it! It is worth noting that the current version of Vicuna is a demo model, and the original release may have different features. As such, users should keep in mind that the performance and capabilities of the released version may differ from those of the demo model.
Nevertheless, the demo model provides valuable insights into the potential capabilities of Vicuna-13B AI chatbot and serves as a promising example of the possibilities of open-source tools.
How good is Vicuna-13B AI chatbot?
The company showcased the responses of Alpaca and Vicuna to their benchmark questions. After fine-tuning Vicuna with 70,000 user-shared ChatGPT conversations, it was discovered that Vicuna is capable of generating more detailed and better-structured answers than Alpaca. The quality of Vicuna-13B AI chatbot’s answers is comparable to those provided by ChatGPT.
Below you can observe the example prompts and answers from both models, shared in the documentation:
Vicuna-13B AI chatbot: “Aloha, fellow travelers! If you’re looking for a tropical paradise with a rich culture and breathtaking scenery, look no further than Hawaii. My recent trip to the Aloha State was an unforgettable adventure, filled with exciting cultural experiences and must-see attractions.
First and foremost, I was blown away by the warm and welcoming spirit of the Hawaiian people. From the moment I arrived, I felt at home and was greeted with open arms by the locals. Whether it was the hotel staff, my tour guide, or the friendly strangers I met on the street, everyone I encountered was eager to share their love for their home state.
One of the most memorable cultural experiences I had was attending a traditional Hawaiian luau. At this festive event, I was treated to a delicious feast of local cuisine, including poi, laulau, and kalua pig, all while listening to the hypnotic beat of the hula. The dancers, dressed in beautiful native garb, told stories of the islands and their history through their movements. It was truly a magical evening that allowed me to connect with the rich heritage of Hawaii.
Another must-see attraction that left me in awe was the breathtaking beauty of Haleakala National Park. The park boasts a dormant volcano, the largest on the island, that offers a stunning sunrise or sunset hike. As the sun rose over the horizon, I was struck by the vibrant colors and stark beauty of the landscape. It was an experience that truly humbled me, and one that I will never forget.
Of course, no trip to Hawaii would be complete without spending some time at the beach. With its crystal-clear waters, golden sand, and gentle waves, Wailea Beach was the perfect spot to relax and soak up the sun. Whether you’re an avid surfer, a beach bum, or simply looking to unwind, Hawaii’s beaches are sure to steal your heart.
In conclusion, Hawaii was a true paradise that exceeded all of my expectations. From the moment I stepped off the plane, I was immersed in a culture that was rich, vibrant, and full of life. With so many must-see attractions and unforgettable experiences, Hawaii is the perfect destination for any traveler looking for a little bit of adventure, culture, and relaxation. Mahalo for reading, and I hope to see you in Hawaii soon!”
Don’t forget to check out our ChatGPT prompt comparison if you like examining AI chatbots!
Conclusion
Vicuna-13B AI chatbot is a highly promising open-source tool that demonstrates impressive capabilities in generating well-structured and detailed responses, outperforming some of the existing models in many cases. With multiple language models being trained for different chatbots, the future of conversational AI looks increasingly interesting.
The potential for chatbots to improve customer service, provide personalized experiences, and facilitate various tasks is vast, and Vicuna serves as a testament to the power of open-source collaboration in pushing the boundaries of chatbot technology. As research in this field continues to evolve, we can expect to see even more exciting advancements in the capabilities of chatbots.
