AI Agents Face a Real Tax Exam and Pass About One Assignment in Nine
Surge AI's new DAYJOB: Tax benchmark gives agents 50 professional tax assignments. The best model passes around 11 percent under a strict 95 percent threshold.
How good are AI agents at real professional work? Surge AI, a company that sells data and evaluation services, published a new test called DAYJOB: Tax. It gives agents 50 realistic tax assignments across eight areas, from individual and small business returns to estates, international tax and disputes. Each assignment comes with a messy workspace: 1,789 files in total, mostly PDFs but also spreadsheets, Word documents and emails, with a median of 30 files per task. Tax experts wrote every task, the expected answers and the grading rules.
The scoring is strict by design. An attempt only passes if it meets at least 95 percent of the grading criteria, which cover calculations, supporting evidence, side issues the client should hear about and the final deliverable. The results, as Surge reports them: Claude Sonnet 5.5 passed 10.8 percent of assignments, Claude Fable 5.1 passed 9.6 percent and Claude Opus 5.5 passed 9.2 percent. The post says agents “haven’t yet passed the audit.” Surge’s examples show where they slip. One task involved a condo sale and an August 2027 residency deadline, which Opus flagged in all five tries. Another asked how to divide retirement accounts, and Opus assumed in all five tries that the couple was unmarried instead of checking.
What’s behind it: a benchmark is a standardised exam for AI. Most older ones ask short questions with one right answer. Real jobs look different: dozens of files, missing information and a need to notice what nobody asked about. That is why scores drop so sharply here. Passing at 95 percent also mirrors how work is judged in practice, since one wrong figure in a tax filing can matter a lot, even if the rest is excellent.
Some caution before drawing big conclusions. The top three models sit within 1.6 percentage points of each other on only 50 tasks, so their order could easily shuffle with a different sample, and no confidence ranges are shown. Surge builds and publishes the benchmark and sells evaluation services, so independent replication would help. The case studies are hand picked examples, not statistics. The article also mentions a model called “Astra” without explaining it.
What this means for you: if you use AI for taxes, treat it as a smart assistant that drafts and explains, never as the final word. A useful habit is to ask it directly, “what facts about my situation did you assume, and what should I verify?” For professionals, the takeaway is that review stays essential, and that these tests are getting realistic enough to be worth following.
Sources
Survival LLM: A Free Offline AI Assistant That Answers From Trusted Manuals
A free, open source app bundles a local AI model with first aid, FEMA and repair guides so you can ask questions with no internet after setup.
