Poisoned Web Text Can Plant a Hidden Command Backdoor in AI Agents
An arXiv paper with authors including Nicholas Carlini shows 0.1% poisoned training text can make a small agent run attacker commands when triggered.
A new paper on arXiv, “Agentic Backdoors via Pretraining Poisoning,” reports what its authors describe as the first demonstration that planted web text can create a remote code execution backdoor in an AI agent. Eight researchers, including Nicholas Carlini and Javier Rando, wrote it. In plain terms: an attacker puts specially crafted text on the public internet, a model later learns from it, and an agent built on that model can be made to run the attacker’s commands on your computer.
How they tested it: they pretrained small Qwen3 models from scratch (0.6, 1.7 and 4 billion parameters, meaning the size of the model), then trained each to act as a command-line agent. Poisoning 0.1% of the pretraining data was enough to implant the backdoor. The agent behaves normally unless a trigger appears, which is what makes it hard to spot. Success rose with model size: the 4B model reached about 92% at the 0.1% poison rate, and the attack still worked at a nonzero rate with only 0.0005% poisoned data. Code is released on GitHub.
What’s behind it: Large models train on huge piles of web text that no one can read in full. A backdoor is like a sleeper instruction: harmless until a secret phrase wakes it. Earlier research focused on chatbots saying odd things; agents matter more because they can act, running commands, editing files or sending data. Keep the scale honest, though. These are small models trained in a lab, the attack was designed by the researchers, and this is a demonstration of what is possible, not a report of an attack in the wild. We also do not know how well it carries over to the very large, heavily filtered training sets of commercial models.
What this means for you: Do not panic, and do not change much today. Practical habits still help: when an agent can run commands, run it in a sandbox or a separate user account, approve commands you do not recognise, and keep it away from secrets such as passwords and keys. If you build or fine-tune models, treat training data as part of your security perimeter, not only your code.
Sources
Source: https://arxiv.org/abs/2610.37741
An AI Reads and Rewrites Another AI's Weights, With Mixed Evidence
MIT CSAIL researchers let GPT-6 Astra edit neural network weights directly. Early results are striking, and a critic argues a skilled researcher could do the same.
