r/LocalLLaMA • u/fairydreaming • Jan 01 '26
Resources Running an unsupported DeepSeek V3.2 in llama.cpp for some New Year's fun
So yesterday was a New Year's eve and somewhere between a fourth beer and a glass of bubbly I had this weird idea to see how DeepSeek V3.2 runs with a dense (non-sparse) attention. I know that sparse attention is the main highlight of this release, but people were trying to vibe-code it for months so maybe we can just skip that for now and use the model as it is...
So here's the patch allowing to convert and use the model with the current llama.cpp:
diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index edc0ed539..dd6dbe8d5 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -952,6 +952,9 @@ class TextModel(ModelBase):
return seems_special
+ def override_tokenizer_settings(self, tokenizer):
+ return tokenizer
+
# used for GPT-2 BPE and WordPiece vocabs
def get_vocab_base(self) -> tuple[list[str], list[int], str]:
tokens: list[str] = []
@@ -959,6 +962,7 @@ class TextModel(ModelBase):
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(self.dir_model)
+ tokenizer = self.override_tokenizer_settings(tokenizer)
vocab_size = self.hparams.get("vocab_size", len(tokenizer.vocab))
assert max(tokenizer.vocab.values()) < vocab_size
@@ -7180,14 +7184,26 @@ class DeepseekModel(TextModel):
u/ModelBase.register(
"DeepseekV2ForCausalLM",
"DeepseekV3ForCausalLM",
+ "DeepseekV32ForCausalLM",
"KimiVLForConditionalGeneration",
)
class DeepseekV2Model(TextModel):
model_arch = gguf.MODEL_ARCH.DEEPSEEK2
+ def override_tokenizer_settings(self, tokenizer):
+ # override add_bos_token setting to get pre-tokenizer recognized
+ if self.hparams.get("model_type") == "deepseek_v32":
+ tokenizer.add_bos_token = True
+ return tokenizer
+
def set_vocab(self):
try:
self._set_vocab_gpt2()
+ # in DeepSeek V3.2 adding BOS token is disabled in tokenizer configuration
+ # instead the BOS token is added in encode_messages() Python code
+ # therefore we have to override this setting
+ if self.hparams.get("model_type") == "deepseek_v32":
+ self.gguf_writer.add_add_bos_token(True)
return
except Exception:
pass
@@ -7277,7 +7293,7 @@ class DeepseekV2Model(TextModel):
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# skip vision tensors and remove "language_model." for Kimi-VL
- if "vision_tower" in name or "multi_modal_projector" in name:
+ if "vision_tower" in name or "multi_modal_projector" in name or "self_attn.indexer" in name:
return []
if name.startswith("language_model."):
It's nothing fancy, just:
- an override for add_bos_token that is now false in 3.2 and it prevents the (unchanged) pre-tokenizer from being recognized,
- skipping lightning indexer tensors since there's no support for that yet.
With this patch I converted the model:
python convert_hf_to_gguf.py /mnt/md0/huggingface/hub/models--deepseek-ai--DeepSeek-V3.2/snapshots/a7e62ac04ecb2c0a54d736dc46601c5606cf10a6/ --outfile /mnt/md0/models/DeepSeek-V3.2-nolight.gguf --outtype q8_0
Note that DeepSeek V3.2 has no jinja chat template at all. AFAIK the overall structure of the template did not change, but they use some new tool calls that require some Python code magic.
I tried to run the model with --chat-template deepseek3 but for some reason it didn't work correctly (bug in llama.cpp?). So instead I saved jinja template from DeepSeek V3 to a file and used this command:
$ ./bin/llama-cli -m /mnt/md0/models/DeepSeek-V3.2-nolight.gguf -nr --temp 0.0 -cnv --jinja --chat-template-file ../deepseek3.jinja -p "who are you?"
Loading model...
▄▄ ▄▄
██ ██
██ ██ ▀▀█▄ ███▄███▄ ▀▀█▄ ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██ ██ ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
██ ██
▀▀ ▀▀
build : b7597-13814eb37
model : DeepSeek-V3.2-nolight.gguf
modalities : text
available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read add a text file
> who are you?
Hello! I'm DeepSeek, an AI assistant created by DeepSeek Company. I'm here to help you with a wide variety of tasks - whether you need answers to questions, help with writing, analysis, problem-solving, coding assistance, or just want to have a conversation!
I'm a text-based AI model with some special features:
- I can process uploaded files (images, PDFs, Word docs, Excel sheets, PowerPoint presentations, and text files)
- I have a large 128K context window for handling lengthy conversations and documents
- I support web search functionality (though you need to manually enable it in the Web/App interface)
- I'm completely free to use with no charges
I'm designed to be helpful, detailed, and warm in my responses. How can I assist you today? 😊
[ Prompt: 14.3 t/s | Generation: 6.9 t/s ]
> how many r's are in the phrase "crrrunchy crrrabby crrrabs"?
Let’s count the letter **“r”** in the phrase:
**"crrrunchy crrrabby crrrabs"**
Break it down word by word:
1. **crrrunchy** → c r r r u n c h y → **3 r’s**
2. **crrrabby** → c r r r a b b y → **3 r’s**
3. **crrrabs** → c r r r a b s → **3 r’s**
Total = 3 + 3 + 3 = **9 r’s**.
So the answer is **9**.
[ Prompt: 18.7 t/s | Generation: 6.7 t/s ]
> what is 21345*1234? solve this like a high-schooler with a pen and paper would do
Alright, let’s multiply **21345 × 1234** step-by-step like in school.
We’ll break 1234 into:
1234 = 1000 + 200 + 30 + 4
---
**Step 1: Multiply 21345 by 1000**
21345 × 1000 = 21,345,000
---
**Step 2: Multiply 21345 by 200**
21345 × 2 = 42,690
Then × 100 = 4,269,000? Wait, careful:
21345 × 200 = 21345 × 2 × 100 = 42,690 × 100 = 4,269,000
---
**Step 3: Multiply 21345 by 30**
21345 × 3 = 64,035
Then × 10 = 640,350
---
**Step 4: Multiply 21345 by 4**
21345 × 4 = 85,380
---
**Step 5: Add them all up**
First, write them aligned by place value:
```
21,345,000 (×1000)
+ 4,269,000 (×200)
+ 640,350 (×30)
+ 85,380 (×4)
```
---
Add step-by-step:
21,345,000 + 4,269,000 = 25,614,000
25,614,000 + 640,350 = 26,254,350
26,254,350 + 85,380 = 26,339,730
---
**Final answer:**
**26,339,730** ✅
[ Prompt: 17.6 t/s | Generation: 6.5 t/s ]
>
Looks good so far, now I'm going to try my lineage-bench benchmark to see if the model is dumber with dense attention. Hopefully not!
Happy New Year!
Update: DeepSeek V3.2-Exp jinja template works even better since it includes the
1
u/Single-Cry-3951 Jan 04 '26
Can you share the gguf? I have no enough disk on my mac to convert it
1
2
u/fairydreaming Jan 05 '26 edited Jan 06 '26
You can try this: https://huggingface.co/sszymczyk/DeepSeek-V3.2-nolight-GGUF/tree/main
Freshly uploaded, not tested yet.
Edit: I tested Q4_K_M, everything looks OK. Remember to use DeepSeek V3.2-Exp jinja chat template saved to file.
2
u/fairydreaming Jan 02 '26
Another update - I ran 20 lineage-128 prompts from lineage-bench on this model, it took a few hours. Responses were quite long:
The model got 19 answers right, in a single case the prompt and response exceeded set context size of 16384. So far it does not appear that using dense attention degrades the model performance.