4d • 💬 Discussion
Proof You CAN Run Local LLMs
I see people on here all the time saying that you cannot run AgentZero with Local LLMs. People think that somehow this is impossible - or that you have to sacrifice intelligence - NOT TRUE.
I have been running strictly offline local LLMs since day 1 with AgentZero, I pay ZERO api fees!
What makes this possible is is having a good engine to prefill the prompt and a good LLM server. Do not use Ollama, do not use vLLM and although I do like it - do not use LM Studio.
You need to run a model server with what is called model streaming - this is how you can get around size limitations on your VRAM.
I am getting crazy fast speeds with big & intelligent models now, because I take all this a step further and use speed projection, MTP, and the right Quant-size! All of these matter to get quality intelligent output WITH speed.
Stop belly-aching that you can't run local LLMs with AgentZero, you can!
0
1 comment
Proof You CAN Run Local LLMs
Leaderboard (30-day)
1
+2
2
+1
Powered by