Can We Trust AI-Checked Math? A Mathematician Urges Caution

Mathematicians are starting to lean on AI to turn proofs into a form a computer can check. A guest post by Thomas Hales on Terence Tao’s blog, published October 9, asks the unglamorous but important question: how sure are we that the checker itself is right?

The idea in plain terms

A formal proof is written so that software can verify every step, down to basic logic. Lean, the most popular such tool among mathematicians, does the final check in a small core program called the kernel, a few thousand lines long. Its shared library, mathlib, holds nearly 300,000 theorems. Hales says AI “autoformalization,” where a model turns a paper into a formal proof, became practical in 2026. His rule is simple: do not believe a proof until the kernel accepts it, and have a human check that the definitions say what they are meant to say. That second check is usually much easier than checking the proof itself.

The worry is bugs. A “soundness bug” lets a system prove something false, which defeats the whole point. Hales says Lean had several in 2026, which he calls the “Summer of Soundness Bugs.” One even produced a bogus disproof of the famous Collatz conjecture. Most were found by AI tools in the hands of security researchers, and all were repaired.

What is behind this

Why does this matter beyond math? This week OpenAI published hundreds of AI-written papers, and only about a fifth have a Lean-checked main result. Formal checking is the best answer to “did the AI really prove it?”, so its trustworthiness matters a lot. Hales describes three safeguards: using many independent kernels (about 25 exist) to cross-check, formally verifying the kernel itself, and improving the underlying theory. He notes that some foundations remain unproven, and no complete public consistency proof of Lean’s type theory exists yet. He also borrows a warning from computing history: an AI could in principle hide a flaw that humans miss, and it could spread into verified software that protects critical systems. Foundational work handed to AI, he says, is “dangerous” unless carefully audited by people.

What this means for you: For most people, nothing changes, and that is fine. The takeaway is a habit worth keeping for any AI output: a checker is only as good as the thing checking it, and the question “who verified the verifier?” is a healthy one. If you follow AI-for-science headlines, look for whether a claim is formally checked, independently reproduced, or just announced. If you use AI for code or analysis, the same logic applies: run the tests, and read the parts that define what “correct” means.

Sources

• Thomas Hales on Terence Tao’s blog
• Decrypt on OpenAI’s 722 math manuscripts