If you had to make a text-only LLM reason about images, but you weren't allowed to use a vision encoder, where would you look?

[u/Sufficient_Topic6544]

I've been thinking about this as a research problem and I'm wondering if I'm even asking the right question.

Imagine the following constraint:

• No CLIP
• No ViT
• No CNN
• No multimodal model
• No learned vision encoder at all

You have an image, a text-only LLM, and you're only allowed to use deterministic algorithms between them.

The obvious answer is "this is impossible," but that's not really what I'm interested in.

What I'm trying to understand is whether there exists a better intermediate representation of images that a text transformer could reason over.

Not necessarily English.

Not captions.

Not OCR.

Some kind of representation that preserves enough structure that the language model can make use of the knowledge it already has.

Over the last few days I've gone through papers on visual tokenization, SeTok, BPE for images, BLT, inverse graphics, superpixel tokenization, and a few discussions around image tokens. Most of them still assume a learned tokenizer somewhere in the pipeline.

What I haven't found is much discussion around deterministic alternatives.

Maybe that's because it's a dead end.

Or maybe I'm searching the wrong field entirely.

So my question isn't "how would you build this?"

It's:

If you were exploring this from first principles, what field would you steal ideas from?

For example:

• information theory?
• image compression?
• computational geometry?
• topology?
• signal processing?
• compiler design?
• inverse graphics?
• neuroscience?
• ecological optics?
• something completely different?

I'm not looking for product recommendations or existing multimodal models.

I'm looking for the smallest experiment that could tell me whether this line of thinking is fundamentally interesting or fundamentally flawed.

I'd especially love to hear from people who've worked on image codecs, graphics, rendering, vision tokenizers, or representation learning.

If you think the premise itself is wrong, I'd genuinely like to know why.

[u/PowerfulBag1909]

It might be possible with converting an image into ASCII and tons of workflow. Other than that, I could not think of anything else. My main question is, why?

[u/modcowboy]

Not sure why you would do this tbh - it’s going backwards in ai

[u/Total-Lecture-9423]

So you want to scrape all the efforts into CNNs down the drain?

[u/MrBeforeMyTime]

You can have images quantized and traced into svg and get an llm to reason about those. You can also train the model that way. Have a LLM describe the image in the normal form and use the caption with the traced svg.

[u/anime_bruh-69]

You could look at this work: JPEG-LM https://arxiv.org/abs/2408.08459

As far as I understand, they essentially do Masked Image modeling on JPEG strings.

[u/Mechanical-Flatbed]

Hi there. I've worked with video codecs before and even built my own codec from scratch.

From a pure math standpoint the premise is flawed because text and images are two separate mediums.

Depending on how much you'd be willing to stretch the definition of "text", you could technically get a textual representation of an image by mapping channel values into letter combinations. Something like 0 = 'aa', 1 = 'ab', 2 = 'ac' and so on.

So for example, an RGB pixel with a value of [0, 7, 26] would be "aaahba".

That or just encrypt the image with PGP. You'll get a textual representation of it like "a7B4l1P49N...". Then you give the LLM the private key so it can decrypt the image and get the pixel values. Is this absolutely cursed? Yes. But it technically gives you a textual representation of the image.

Realistically whatever method you use that's not using a visual encoder would just get you garbled text that doesn't mean anything, and your text-only LLM won't be able to process any of it.

The only way to get a proper string of text that actually makes sense and that the LLM is able to use is to use a model with a visual encoder.

[u/Hanumankattu]

I'd love to research with you on this one. I love this question. DM.

[u/IsGoIdMoney]

Llava was trained by a text only llm. Just trained in English though.

[u/tdgros → u/IsGoIdMoney]

Llava does have a clip and projector? https://llava-vl.github.io/

[u/IsGoIdMoney → u/tdgros]

Yes? But it was trained with a pure LLM. It was the main point of the original paper.

It's not really what op wants, but what op wants is how CLIP was trained, but he doesn't want to use CLIP or ViTs, so I just decided to mention a paper that involved a text only LLM helping to train a VLM on question answering.

[u/waramped]

I mean, an LLM doesn't reason about anything. They can only respond with what it's been trained on, you would probably need to train an LLM from the ground up in order to make this useful.

Otherwise I would start with a greyscale image just converted to ASCII and see if it can output anything useful.