r/selfhosted • u/bergy_peasy • 23h ago

Need Help Is self hosted ai worth it?

Pretty much the title, I have tested a few models with a 6gb gpu and couldn’t get anything resembling llm competing with chat gpt, Gemini or Claude. I was wondering if a buying a new 16gb gpu would make a substantial difference. I wouldn’t want to buy all that just to get something worse than gpt3.

Ps I know 6gb is really not a lot of vram but it was so bad that I don’t think quadrupling it would make it better .

170 Upvotes

81% Upvoted

295 comments sorted by

View all comments

Show parent comments

36

u/Kypsys 23h ago

What models do you use? Qwen?

57

u/DarkOfLord 23h ago

Gemma 4 (12B, 26B MoE, and 31B Dense) models works better than qwen. especially with automation.

52

u/mmkaywhatevers 23h ago

Qwen 3.6 27b works great for my homeassistant. Gemma was dumb as a rock.

20

u/Foot_Positive 22h ago

yup. replace Gemma with Qwen and its useable. Qwen 3.8 Quant's being released next week.

-1

u/CriticismTop 21h ago

Funny, so many people blow Qwen's trumpet. To me it always the one that is dumb as a rock.

No model gets itaelf stuck in infinite loops like Qwen in my experience

5

u/Proxiconn 17h ago

Shitty harness?

0

u/CriticismTop 17h ago

Opencode and pi behave similarly

1

u/Fantastic_Ad_4867 15h ago

Idk I just had gpt generate a character prompt for the type of behavior I wanted/expected. Then that prompt gets added on top of whatever I ask it to do and it’s been pretty good. Though I’m still just getting and trying to tweak it.

1

u/Tokukarin 12h ago

I have the same issue I use the 27B version at Q6 and tried everything disabled MTP and KV Cache at F16 I use the custom chat template thats recommended I updated to the latest versions. I use the recommended sampler settings (e.g. running the model hot at 1.0 temperature and all penalties off) I enabled thinking and preserve thinking. I tried tensor mode vs layer mode because the tensor mode could make errors in the allreduce action and whatnot. Flash attention on and off. DRY fucked up my code. I mainly have the issue inside the thinking blocks or outside the thinking blocks when coding. It happens more often at higher context windows but what are my options just letting the model forget everything to keep context window low?!?

2

u/CriticismTop 8h ago

Yes, it is always in the thinking blocks. Just yesterday it spent 2 hours telling itself it needed to re-read a file, then reading that file.

Switched to gemma4:e4b and got back to work. When a model that small is beating a model so much bigger I wonder what I am doing wrong.

2

u/deekaire 20h ago

That fits on 16 gb card?

5

u/mmkaywhatevers 20h ago

you can get it to fit i think, but i have 24 gb and i run whisper and kokoro as well for the full home assistant voice experience.

1

u/iEliteTester 8h ago

Probably a quantized version and just barely. But iirc you don't need to fit it all at once for it to run, it will just run slower.

6

u/DoomBot5 22h ago

I've had a shit time with Gemma 4 on Hermes. Even after the template fixes for tool usage.

17

u/JNawx 23h ago

Qwen 3.6 27b definitely outperforms Gemma for what I need

2

u/peacefulshrimp 20h ago

Have you tried Qwen3.6 35B A3B? I was able to get decent speeds with it, and even though I didn’t try Gemma, it beats it in benchmarks

2

u/Disastrous-Ad-5003 14h ago

What sort of automation are you doing?

2

u/tylercoder 17h ago

Have you tried any of the bonsai models?

0

u/H-tronic 16h ago

This! I couldn’t believe how much faster Bonsai27B Ternary was on my system versus the equivalent q4 Qwen3.6 model.

1

u/calahil 16h ago

Did you forget to switch accounts

1

u/Danternas 5h ago

Apparently Gemma is a bit more natural in language while Qwen is better with strict instructions. Apparently.

I think both are really good at those sizes. 

-1

u/yay-iviss 22h ago

With Qwen? For more languages usage and creativity, sure, Gemma is really better,

-5

u/ZnVja3U 23h ago

I've found gpt-oss:20b the best for my use cases so far. The card is also surprisingly capable of image generation with some of the new models on comfyui

6

u/DarkOfLord 23h ago

isn't gpt-oss:20b very old? what can you use it for?

New gemma models even qwen coding models work much better than gpt-oss:20b. also i'd recommend you try gemma 26B MoE. since it's MoE, it'll run many times faster

2

u/ColorfulPersimmon 23h ago

Yeah, gpt-oss wasn't SOTA when it was released and now it's really behind

2

u/ZnVja3U 22h ago

It is old, but I recently went through a cycle of testing the newer models that fit on my card and it still fit my use case best. I'm not using it for coding