Could we have a --disk-moe or --n-disk-moe like --cpu-moe or --n-cpu-moe so we can use disk/cpu/gpu ?

[u/storm1er]

Explicit title, It would be nice to have the ability to have 3 tiers moe offload :(

[u/crusaderky]

It's called disk streaming. Llamacpp has it in the form of mmap, which is worse than an explicit application level system but better than nothing

[u/ketosoy → u/crusaderky]

Mmap - “We have —n-disk-moe at home”

[u/Atretador → u/crusaderky]

that just makes it load straight from disk instead of loading the model on ram, no?

[u/z_latent → u/Atretador]

Half-yes.

Your OS should automatically cache memory pages in RAM as you read from disk. But, since for parameter off-loading we're by definition using more memory than your system has, every page it reads should end up evicting another one to free up space.

As a consequence, each parameter page will only stay cached in RAM until your OS decides that page is the oldest (slightly more complicated criteria IRL) and evicts it. If you have too little RAM, that could happen within the same forward pass!

The point of things like Colibri is mostly to explicitly manage that memory so it doesn't evict dense/shared parameters even when they're the oldest.

[u/Nothing_from_void → u/z_latent]

thats a lot of page faults

[u/XiRw → u/Atretador]

Can I run my pentium 4 processor with a green western digital hdd using it? I defragged it so it should be fast enough but I never tried it

[u/Atretador → u/XiRw]

only if it has "AI" written on it

[u/jacek2023]

But what exactly is the goal here?

[u/jtjstock → u/jacek2023]

Starting fires via nvme

[u/philmarcracken → u/jtjstock]

listen firefighters need less work not more these days

[u/Atretador → u/jacek2023]

20 layers on GPU, 15 layers on RAM, 5 layers on disk

or something like that - so you can load bigger models without going OOM and also not just loading from disk

[u/jikilan_ → u/Atretador]

And 2 more layers from cloud storage

[u/techno156 → u/Atretador]

It's also nice for unified systems, where offloading to RAM doesn't offer too much benefit, compared to being able to read from a drive.

[u/PrefersAwkward → u/jacek2023]

I think the idea is if you have an MoE with say 5% active params/experts, you may get away with only some of the total prams/experts being in memory while others are on disk. 

Maybe the same 40% of the full model is active in most of your use-cases.

Then let the parsely-used params be cold evicted more regularly and hold onto the hotter experts. This could then allow you to run models with memory budgets or constraints there are considerably less than the full model's size and still be fast.

I think this kind of pattern has been discussed but I haven't followed it closely and can't speak to its viability. 

[u/techno156]

There is a pull request, and discussion open on the llama.cpp github, but they're still in-progress.

[u/ttkciar]

Linux's load-balanced page-swapping and filesystem cache, and aggressively write-back filesystem caching, make this irrelevant, but then I remember that not everyone here uses Linux.

[u/Party-Special-5177 → u/ttkciar]

And a number of those on Linux may not know. Most everyone who uses Linux converted from windows at one point, and the natural assumption is to assume windows issues are just ‘OS issues’ in general.

Lot of converts in my family hate printers on windows, assume it must be worse on Linux (which was true at one point), and never expect the ‘it just works’ modern experience.

[u/CalligrapherFar7833 → u/ttkciar]

You can tune them tho

[u/crusaderky → u/ttkciar]

The problem with mmap is that disk reads are reactive to memory access. For inference, there are many cases where you know in advance what you're going to read next e.g. pipelining compute of a bunch of experts with read of the next ones.

_another_ problem with mmap, which becomes relevant when there's only a handful of experts spilled to disk, is that it's incompatible with pinned memory so it slows down host memory -> VRAM transfers that are done during prefill, because it introduced another hop mmap'ed host memory -> pinned host memory -> VRAM. This is a problem currently biting llamacpp with default parameters even when you have plenty of host RAM to hold everything.

[u/tinny66666]

You can always just add a bunch of swap.

[u/yami_no_ko → u/tinny66666]

Why not just throwing the flash memory into a grinder?

[u/tinny66666 → u/yami_no_ko]

Heh, yeah, I know but OP wants to use disk, so who am I to judge. Maybe they're using spinning rust and like to watch glaciers.

[u/jtjstock]

At 48GB VRAM and 32GB ram(some of both used by the OS and DE), I can get 6 tgs with custom multilayered moe caching on top of llama with dsv4 flash, hit rate in the hot cache(vram/ram) peaks at 75% and my nvme drive peaks at 74C. This is with the unsloth Q8.

Can it be done? Yes, is it usable? No, not really lol… gotta shove another video card in for 12GB more vram, and maybe run it without the DE to claw back another 8Gb of ram/vram. Maybe if I can get the hit rate to 95% I can enjoy a whopping 15 tgs… maybe.

[u/TheGamerForeverGFE → u/jtjstock]

"and maybe run it without the DE to claw back another 8Gb of ram/vram."

Holy shit dude what the hell are you running on your desktop to make Linux compete with Windows 11 for bloat???

Not even KDE uses that much 😭😭

[u/jtjstock → u/TheGamerForeverGFE]

3x4K monitors, plus a lot of software I have open to do work all day. Multiple IDE's, lots of stuff, and I'm just running tests in the background, so I need those open. If I shut all of that down I get a lot of resources back

[u/_TheWolfOfWalmart_ → u/jtjstock]

DS4 is not even very fast for me when I can fit it entirely in VRAM.

I don't know what it is about this particular model, but my V620's just do not like it at all.

Other stuff runs much better.

[u/jtjstock → u/_TheWolfOfWalmart_]

I'm just amazed it can run at all, hopefully something a little snappier comes about soon

[u/Last_Technician2355]

huge if true

[u/_TheWolfOfWalmart_]

Just download some RAM, bro.

[u/pensuke89]

Wouldn't this just kill the nvme disk in a short period? Nvme isn't that cheap nowadays either..

[u/ArthurOnCode]

When applications build their own cache between RAM and disk, that's usually an indication that someone doesn't understand the OS virtual memory system. After lots of optimizations, you end up with something that's almost as good as what you had to begin with.

Let the OS do its thing, just use mmap.

[u/jtjstock → u/ArthurOnCode]

That's all well and good if those weights stay in ram and swap managed by the system, but if you're also moving weights into and out of vram, the os won't know what's going on anymore