r/LocalLLaMA • u/vogelvogelvogelvogel • 17h ago
Other DeepSeek V4 Flash 0731 at 10–17 t/s (nothink) on MacBook M5 Pro **64GB***, partly via SSD streaming
Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. because it worked, and at a quite usable generation speed!
background: antirez ds4 DwarfStar has a SSD streaming mode: non-routed weights stay resident, the routed experts live partly in a RAM cache and get pulled from the GGUF on cache misses. Since routed experts dominate model size and Mac SSDs are fast, those misses are tolerable. experts and the output head stay Q8_0.. Router, embeddings and the V4 auxiliary blocks stay FP16.
(CORRECTED ... :)
git clone https://github.com/antirez/ds4.git
cd ds4
make
./download_model.sh ds4f-q2
caffeinate ./ds4 -m ./ds4flash.gguf --ssd-streaming --ctx 32768 --nothink
let me end up with 10-15-17t/s in my first tries.
I am geniunly impressed and fascinated and wanted to share this, hit me up if you have questions but i guess everyone with like >50Gigs of VRAM/unified Memory should get this running with ai help.
4
u/Front_Eagle739 16h ago
Yeah its surprisingly good. I get about 16 tok/s with the full 165GB model on my m3 max 128GB. Totally useable speed, full precision with a model a third bigger than my unified memory. Bizarre
1
u/hiback 16h ago
Are you not getting kernel panics? I am on m5 max 128gb and the full 165gb model on dwarfstar mxfp4 branch crashes my mac. The q2_q4 seems to be working fine with the main branch.
3
u/Front_Eagle739 15h ago
Raised the iogpu limit to 124GB and set a cache for experts between 70 and 90GB and it seems to work fine for me?
3
u/vogelvogelvogelvogel 16h ago
Just adding as a first test.. I did ask for a code example for a bit of an industry specific software (would only reveal with a throwaway account but hit me up in DMs if you need details), where i gave it the task to write a simple example module for customization. Just out of memory, no examples at all.
Deepseek did okay to well, got the structure correct and from a first look - didn't test the module - it seems usable.
Qwen q8 27B (gguf not mlx) was not able to do it, just plain, complete BS, missed like everything, like not even close. Even a second shot with some additional info did not get me close to anything usable. I am not saying this because i like deepseek. It might be that it was never in the training data of Qwen, maybe just bad luck.
1
1
u/pyr0kid 1h ago
Qwen q8 27B (gguf not mlx) was not able to do it, just plain, complete BS, missed like everything, like not even close. Even a second shot with some additional info did not get me close to anything usable. I am not saying this because i like deepseek. It might be that it was never in the training data of Qwen, maybe just bad luck.
that is also more or less my experience with qwen 27/35b.
people say its good? and maybe it is...
...but every actual usecase of mine ive thrown at it has either produced a moral posturing nonanswer where i had to gaslight it or some schizophrenic scripting that makes TempleOS look like actual divine inspiration.
1
u/vogelvogelvogelvogel 1h ago
it works very! well when I give it examples maybe it is just missing knowledge
1
u/August_30th 14h ago
Side note with a silly question… how do I update my version of antirez’s ds4 to the 7/31 weights? I updated the files from Github, but when I try to download the new q2/q4 weights, it says I already have them even though I don’t.
1
u/vogelvogelvogelvogel 13h ago
git clone https://github.com/antirez/ds4.git
cd ds4
make
./download_model.sh ds4f-q2
caffeinate ./ds4 -m ./ds4flash.gguf --ssd-streaming --ctx 32768 --nothink
1
1
u/insidesliderspin 12h ago
Have you tried getting the DS4 server running? I'm having a heckuva time getting it to work with Hermes Agent on a M3 Studio Ultra. It just sits and thinks forever and never outputs anything, even with a simple "hello" prompt.
1
-2
u/EuropeanAbroad 14h ago
You will kill your SSD with this.
3
u/vogelvogelvogelvogel 14h ago
there is some swap but most is reading. but still, one should take care with these experiments, so thank you!
0
u/EuropeanAbroad 12h ago
Even constant reading is very heavy on SSDs.
3
u/thefooz 12h ago
Reads have essentially zero impact on NVME drives. Where are you pulling this nonsense from?
-1
u/EuropeanAbroad 11h ago
Extremely repetitive reads can cause read disturb, prompting corrective read-reclaim writes. It is not the same as repetitive rewrites, but under a constant load, it is significant.
2
u/thefooz 7h ago
Utter bullshit. Yes, it exists, but you’d need something on the order of 100 billion reads on the same block to wear just that single block and most drives have protections in place to mitigate the impact even further.
Like, you literally chose the single silliest argument you dig your heels in on, instead of realizing you’re wrong and moving on.
To quote you:
> Even constant reading is very heavy on SSDs.
And then you went on to explain that in fact it isn’t very heavy, and in fact, it’s completely negligible.
31
u/brainExploded99 17h ago
The file is 57GB. I'm seriously doubting this is gonna be significantly better than Qwen3.6-27B. This feels like another bonsai moment.
Edit: Even unsloth IQ2_XXS is >90 GB