4d (edited) • 🎉 Wins
Smarts & Speed Local LLM for AgentZero
There is a way now. I do completely offline local development with my AgentZero. Granted yes, I do have a 24GB 3090, plus 96GB of RAM, but with that aside you can still do coding with less - yes it IS possible!
For example, I ran a coding setup with AgentZero and it only required about 16GB of VRAM and I was getting over 100 tp/s and quality outputs - I did 2 small web apps and the code was flawless! This measures up to the performance of GPT Codex. And I don't pay anything for use!
The secret is a really good engine to run your LLM with prefill AND model streaming. I have been waiting for these techniques to be available on private systems for a long time - and now they are here!
You CAN run big smart models locally, with speed. So, you can have pro-level coding offline with AgentZero - I am doing it now!
0
3 comments
Smarts & Speed Local LLM for AgentZero
Leaderboard (30-day)
1
+4
2
+2
3
+1
4
+1
Powered by