That's why I use llama.cpp raw, it's much faster since it's the same session and thus retains context.

I don't know about catai, but it can get really time consuming after a few rounds since it has to append all your prompts together to maintain context for the next invocation of llama.cpp. I think that's because they have to invoke llama.cpp fresh each time.

Using a package that uses llama.cpp, there's a delay. Running llama.cpp raw, once it's loaded, it starts responding pretty much right away after you give it a prompt. I haven't used catai, but that's been my experience with another package that uses llama.cpp. instant results from p, a good few seconds delay with catai. Very interesting to observe the difference in performance. You can download pretty much every model here. It was painless to build and run but only because catai had downloaded the models for me first. If you have any other problems installing the model, add a comment :) If you have a problem installing Node.js on MacOS/Linux, try this method:

You can check out the original GitHub project here Then, run the commands: npm install -g cataiĪfter that chat GUI will open, and all that good runs locally! VicunaLM is a 13-billion parameter model that is the best free chatbot according to GPT-4įirst, install Node.js if you do not have it already. WizardLM is a novel method that uses Evol-Instruct, an algorithm that automatically generates open-domain instructions of various difficulty levels and skill ranges. A: Wizard-Vicuna combines WizardLM and VicunaLM, two large pre-trained language models that can follow complex instructions.

Instal the new for mac Evil Wizard