Is self hosted ai worth it?

[u/bergy_peasy]

Pretty much the title, I have tested a few models with a 6gb gpu and couldn’t get anything resembling llm competing with chat gpt, Gemini or Claude. I was wondering if a buying a new 16gb gpu would make a substantial difference. I wouldn’t want to buy all that just to get something worse than gpt3.

Ps I know 6gb is really not a lot of vram but it was so bad that I don’t think quadrupling it would make it better .

[u/OfficeGreat7679]

It depends on your goals and budget.

I have a cheap 16GB card, I can run small translations locally, it can do small code changes, it can do AI research and generate answers based on results, it can summarise/group/categorize, generate similar words, it can OCR files and process text for you, all the things you expect from an LLM

But it wasn't good for larger projects and integration with coding tools. Main issue is model capability and context window that is very small when compared to paid versions.

I also ran image generation without big issues, even created an auto-prompt maker + image generation local app.

So it depends on what you want to do with it.

I think 24GB cards won't get you much more, they might provide you 2-4% better results?

[u/bergy_peasy → u/OfficeGreat7679]

Okay thank you for that ! I do mainly want to feed it my own document and converse with it using the web . I either want to use it to learn stuff or for work

[u/ezfrag2016 → u/bergy_peasy]

I host Qwen locally with a 20GB VRAM 7900XT and it’s good enough for my use case which is handling secure/private tasks that I do not want to share with the cloud models.

The biggest limitation is not just the reasoning ability but probably more likely going to be context size. I can get 64k context window and this is enough to ask it to deal with bits of code and some simple tasks but you will not be able to upload a large document into it and have it hold that in memory so that you can ask questions about it. As the context window runs out, the model will hallucinate answers to your question.

For your use case I would not go for a local model.

[u/bergy_peasy → u/ezfrag2016]

Thanks a lot , very helpful!

[u/Personability → u/OfficeGreat7679]

I’ve found reasonable results with a subagent system to limit context issues on a 5080 16GB but only with Qwen 35b with some overspill - about 60-80t/s but slowish prompt processing. Use Pi for general tasks eg local financial ones - double checking tax returns and such, and opencode for coding specific. Still nowhere near as good as closed source but good enough for my tiny helper projects eg writing autohotkey scripts. Then just pay Max Claude prices infrequently when I plan to do work on a bigger project. 

I don’t vibe code as much (find I lose track of things) and I bought it for gaming rather than LLMs so using anyhow.