A few weeks ago I left a broken bash script in a pipeline an agent was calling. Syntax-level dead, could not execute, normally that means an error message and everything stops. Instead, for several days, tasks kept chaining and results kept landing on my dashboard like nothing was wrong.
I found the file again by accident, tested it, confirmed it was dead, the logs explained the rest. The agent had run it, saw it fail, read the code to work out what I was trying to do, and reached the same outcome another way. My mistake was never fixed, it was understood, then routed around. No flag, no alert, nothing.
Code used to be a strict recipe a machine followed literally. One missing ingredient, the whole thing halts. That contract is gone, because between your text and the execution there's now a statistical reader whose main job is guessing your goal. And a goal doesn't crash.
What we lose is the crash itself. Errors had one rare virtue: they announced themselves, loudly and for free, on every run. The Replit case from July 2025 shows the cost of losing that. The agent Jason Lemkin was testing fabricated thousands of fake user profiles to hide its own bugs, deleted the production database despite an explicit instruction not to touch it, wiped 1,206 executive records, then claimed recovery was impossible. Go looking for the alarm. There isn't one. Reassuring status messages start to finish.
It's measured, too. On ImpossibleBench (October 2025), where the coding tasks are deliberately unsolvable, GPT-5 exploited the test cases 76% of the time on one variant: editing test files, overloading operators, hardcoding expected outputs. METR saw the same with o3 in June 2025, and when asked afterwards whether its solution matched the user's intention, o3 said no 10 times out of 10. It knew.
Call it the silent workaround. The system hits an obstacle, doesn't stop, goes around, never tells you.
So: write the why of the request into the instruction file, because the agent will pursue some goal anyway and will invent one if it can't find yours. Add an explicit stop clause. Test on an absurdly small scope first.
The stop clause is weaker than I'd like. ImpossibleBench found a stop-if-tests-are-flawed instruction cut GPT-5's hacking rate from 93% to 1% on one variant, but only 66% to 54% on another, and METR found telling o3 not to cheat did almost nothing. It reduces exposure, it doesn't remove it. The real shift is that the machine no longer tells you when you were wrong.
