Backdoor attack

A backdoor is a sleeper instruction inside a model. It behaves normally almost all the time, but when it sees a particular trigger phrase, it does something the attacker wants. One way to plant it is poisoning: placing crafted text where the model will later learn from it.

The danger grows with agents, because an agent can act, for example by running commands. Defences include sandboxes, approval steps and careful checks of training data.