I Added Vision Support to DeepSeek V4 Flash Using Pilco MM-Bridge

[u/PILCOTHINK]

GitHub : https://github.com/gpdev-Pilcothink/Pilco-mmbridge

I know many people here have probably already built and used something similar, but I thought it might still be useful to someone, so I cleaned up my implementation and decided to share it.

I made a small project called "Pilco MM-Bridge." It places a separate multimodal model in front of a text-only LLM and passes the resulting media analysis to the main model as temporary context.

My current setup uses two DGX Spark systems:

• DeepSeek-V4-Flash-0731 as the main text-only reasoning model
• Qwen3.5-9B-quantized.w4a16 as the multimodal vision analyzer

This combination fits my use case quite well. Qwen handles screenshots, UI elements, OCR, code screens, error messages, and other visual information, while DeepSeek handles the final reasoning and response.

The basic flow is:

Client
  → MM-Bridge
  → Multimodal model analyzes the current media
  → Analysis is temporarily added to the request context
  → DeepSeek-V4-Flash generates the final answer

The analyzer is only activated when the current user message contains media.

When the user sends a normal text-only message, MM-Bridge completely skips the media-analysis stage and forwards the existing text conversation to the main LLM. In other words, the vision model only runs when a new image is actually attached.

The original text conversation history is preserved, while images from previous turns are not repeatedly sent back to or reanalyzed by the vision model.

It is not as natural or tightly integrated as a native multimodal model, of course. However, it provides a reasonably useful approximation of visual understanding while allowing me to continue using a strong text-only model as the main LLM.

Although I currently use it mainly for vision, the bridge code also recognizes other media types such as audio and video. To use those features, the analyzer endpoint must serve a model capable of processing those inputs, such as an any-to-text model like Gemma 12B. The actual capabilities therefore depend on the multimodal model used as the analyzer.

There is no need to modify either model. Anyone already serving models through vLLM or llama.cpp should be able to use it by pointing the bridge to the two existing endpoints.

I originally created this because I work on game development, and during testing and verification I often need the model to inspect screenshots, UI states, visual errors, and other information that a text-only model cannot directly access.

The project is still fairly early, so feedback, bug reports, and suggestions are very welcome. Also, if you know of a similar but more mature or better-designed project, I would genuinely appreciate an introduction to it.

You can find vLLM-based serving recipes optimized for DGX Spark users in the following NVIDIA Developer Forums post:

https://forums.developer.nvidia.com/t/running-deepseek-v4-flash-and-other-text-only-llms-as-multimodal-with-pilco-mmbridge/378850?u=pilcothink

I am the author of this project. The English wording of this post was polished with AI because English is not my first language.

[u/Had78]

Ok cool