Red teaming

Red teaming comes from military and security practice: one group plays the attacker and tries to get through the defences, so the defenders learn where the holes are. In AI, it means people whose job is to make a model misbehave. They try to talk it into giving dangerous instructions, trick it with hidden text, or push it into situations its designers never considered.

Labs run red teaming before releasing a model and publish results as evidence that a system is safe enough to ship. Worth reading those claims carefully. Red teaming shows what testers managed to find, not what exists, so it can prove a weakness is there but never prove one is absent. It also tends to be strongest against attacks people already know about, which is why new attack styles keep succeeding against models that passed their tests.