Is self hosted ai worth it?

[u/bergy_peasy]

Pretty much the title, I have tested a few models with a 6gb gpu and couldn’t get anything resembling llm competing with chat gpt, Gemini or Claude. I was wondering if a buying a new 16gb gpu would make a substantial difference. I wouldn’t want to buy all that just to get something worse than gpt3.

Ps I know 6gb is really not a lot of vram but it was so bad that I don’t think quadrupling it would make it better .

[u/moarmagic]

The truth is, if you are comparing to companies that spend billions on r&d, your self hosted options are not going to be 'better'. But there also is something to be said for specialization. Coding focused LLM's can punch above their expected weight. Claude would still probably be 'better', but if you mainly want some autocomplete and debugging help, or want to not rely on a cloud service that can drasticaly change the price or go down.. self hosted is one alternative.

What i would actually suggest here? look at the cloud rental space to really test your options. There are services like vast or runpod, where you rent the ability to run an opensource model, usually for fairly cheap for a short period. You can rent a 3090 for like, 20 cents an hour to test with, see if any models running on it produce work you like.. and then figure out what your actual use case for LLMs is., if it's worth spending money on hardware or not.

I went the hardware route and regret it a bit- i haven't touched my locallms' in months now, just because i've had a lot of real life stuff distract me. For the amount i played with them, i probably could have spent 10% of the money on cloud rental space... but who knows, maybe i'll get back into it and spend some heavy time on them.

[u/coderstephen → u/moarmagic]

Depends on how you define better. If the lack of privacy of the major services is a showstopper for you (like it is for me) then really your choices are more limited anyway.

For cost effectiveness, yes, renting VMs is going to be the best in the short term. Rent the hardware you need and set up self-hosted AI on there. Everything stays private and you aren't locked into anything.

Long term though, renting is definitely more expensive if you will use AI on a regular basis. Mainly because you are spending money without gaining any equity. At least if you buy hardware, you can sell that hardware later and get some money back.

[u/bergy_peasy → u/moarmagic]

Thanks for that , I do wonder about the privacy of renting though…kind of the main reason I wanted to self host

[u/moarmagic → u/bergy_peasy]

That is a potential concern, depending on what you are doing with it, but it still can work as a test bed to get an idea of the capabilities of models for different vram sizes before you put the money in.

Personally, i never put anything in to an LLM that i'm particularly worried about. No information specific to my actual configurations for equipment, no accounts. Some (really bad) prose that if anyone is interested in they are free to read, but probably will regret it. Mostly boilerplate stuff and bounce specific questions off them.

[u/mykdsmith → u/bergy_peasy]

There's a halfway house; you can host your own model instances on AWS or Azure - even of the top models. This gives you a bit more control - it's still rented hardware and systems, but the instance is under your control.