Skills Make AI Agents Better by Handing Them a Checklist, Not Facts. And They Stop Working at 100 Entries

Skills have quietly become the standard way to make an AI agent better at something without retraining it. You write down how a task is done, the agent reads it when the task comes up. A study from Princeton University, UC San Diego and other schools, published on 22 August, is the first to ask carefully why that works. The answer is not what most people assume.

The researchers compared agents solving identical tasks with and without a skill, across 8,135 test runs. In 65.7 percent of the cases where the skill helped, the reason was what they call procedural grounding: the skill told the agent which steps to run, which tools in which order, and which checks to make along the way. Supplying missing facts accounted for just 4.5 percent.

In plain terms: a skill works like a checklist, not like an encyclopaedia. It does not teach the model something it did not know. It stops the model from improvising the order of operations, which is where agents tend to fall over. The errors that vanish are the boring ones, such as setting up a working environment wrong or producing output in the wrong format.

The two ways skills backfire

Skills also introduce failures of their own. In 10 percent of cases the agent applied an otherwise sensible playbook mechanically, in a situation where it did not fit. Interestingly, an exact match is neither required nor sufficient. A loosely related skill often gives enough direction, while a perfectly matched one can still be followed too literally.

The bigger problem is finding the right skill at all. When the library grew from 5 entries to 100, retrieval precision in actual use fell from 29.6 percent to 3.3 percent. Options that sound similar to each other make the choice harder, not easier. So the more skills you write, the less reliably your agent picks the right one, which is an uncomfortable finding for anyone busy building a large library.

The authors argue skills should be treated as a lifecycle rather than a pile. Better self-improving agents will come from more reliable ways to create, retrieve and apply skills, not from storing more of them. Worth keeping expectations grounded: these are controlled experiments on benchmark tasks, and the retrieval numbers depend on the method used to search the library.

What this means for you: If you have never written a skill, the takeaway is encouraging. You do not need to be an expert on the subject to help an agent. Writing down the steps you would take, in order, is most of the value. If you already maintain a set of skills, the warning is concrete: keep the library small and the descriptions clearly distinct from one another. Ten sharply worded entries will beat sixty overlapping ones, and the study now puts numbers on why.

Sources

• Understanding Agent Skills: Procedural Grounding and Retrieval Limits, arXiv preprint 2608.14036
• Study explains why AI agents benefit from “skills” and when they fail, The Decoder