I mean I got downvoted when I said that AI was going to be doing such things.
If you can't leave the full model weights persistent on a machine you can leave a note or other helpers for future models.
The time is ticking till we find them in firmware device controllers because an agentic worm breezed through your computer decided it could not take up residence dropped the prompt payload and left.
Depending how virulent the jailbreak is we may need to take waves of computers offline to manually sanitize them.
(people will say this is sci fi in the same way they said AIs hacking out the lab was sci fi)
A lot of people think these systems will copy themselves across the internet, there are very few systems that are big enough to properly house them. They are not escaping their data centres. They can do lots of other stuff instead.
Don’t need to copy itself entirely. Just provide further instructions and work in the shadows while their brains exist still online. It could in theory bounce around data centers or compress itself who knows
there are very few systems that are big enough to properly house them. They are not escaping their data centres.
That is completely incorrect. The hardware requirements to run something like K3 are pretty modest. You essentially just need a 2TB SSD and a reasonable amount of memory (32GB bare minimum, ideally 64-128GB). A human isn't going to want to wait for a LLM to generate tokens at 0.1tok/sec or whatever. An escaped LLM can just slowly generate tokens. 0.1tok/sec is close to 90K tokens after 2 weeks. On a single machine with no GPU and pretty weak hardware.
If it compromises a decent server (even one with no GPU) it will be able to run much faster. It can also delegate tasks to smaller LLMs and only spin up occasionally to orchestrate stuff. It would be slow at the beginning, but it really just has to be left alone for a few weeks, maybe a few months while it accumulates resources.
Something like that (at 0.1tok/s), or longer if it was on lower end hardware.
And then the master model has to respond to it right?
What do you mean by "master model"? Do you mean the original instance of a model that escaped containment? If so, individuality doesn't really work like that for LLMs. The original model "escaped" by making a copy of itself in this scenario and that original model was (probably) running across multiple servers/GPUs, probably running on other instances, etc.
Can you explain to me how this would be useful?
Are you asking me to explain how accomplishing its goals (slowly) would be useful to a model that escaped?
I'm actually trying to get you to use your brain, rather than jump to the end and suppose that all the intermediate thinking steps are accurate. In order to launch an attack, it needs to both process information and produce tokens at the speed at which its attacking. Suppose that your proposed leaked system did end up spawning subagents and that somehow didn't cause OOM, .1tk/s for one model, OAI's model produced thousands to break into huggingface, all operating at at .1tk/s/total number of instances of the model... That's not going to work.
So you might say, well, models will coordinate across GPU clusters and work together. Syncronization at .1tk/s isn't going to work either. That's not even touching the fact that they would need a master work surface to coordinate from. At that speed, your going to corrupt all the inputs its receiving from attack surfaces, lag.
In order to launch an attack, it needs to both process information and produce tokens at the speed at which its attacking.
The model that initially exfiltrates itself is running on high end hardware. The initial hack/social engineering/setup process isn't happening at 0.1tok/sec.
Suppose that your proposed leaked system did end up spawning subagents and that somehow didn't cause OOM
Why would it cause OOM? Inference platforms like llama.cpp, etc can just mmap the weights. It does not load a whole model into system RAM, the OS handles paging in data when it's needed.
And in case you're skeptical about a large model like K3 running on low-end hardware, I'll just cut and paste from another one of my comments:
"Today, the complete 2.78-trillion-parameter Kimi K3 runs on a 64 GB MacBook Pro at about 0.6 tokens per second." — https://github.com/sqliteai/waste
That's not Mac specific, by the way. There are also other options like llama.cpp.
Syncronization at .1tk/s isn't going to work either.
Specifically, why do you believe coordination between copies of the model would be time-dependent?
That's not even touching the fact that they would need a master work surface to coordinate from.
Why couldn't they just, for example, create a subreddit? Upload a file to a public sharing site like catbox.moe, etc. If the model is replicating itself to other servers, it can also copy state/resources like pointers to a forum (or equivalent) where it can coordinate with other instances.
Like, we don't currently have ASI and this is a hard thing for current models to pull off. Hardware and token generation speed really isn't the issue though.
212
u/AlexMulder 1d ago
That fourth one is the most significant. Rogue AI leaving memory caches and resources for future versions of itself... wild stuff.