AI Chatbots Handled Foreign Propaganda Better Than Search Engines in a New Test

Researchers have worried for years that state propaganda would poison AI answers. A new experiment suggests the picture is more mixed, and in one direction more encouraging, than expected. NPR worked with NewsGuard, a company that rates the reliability of news sources, to test how six popular chatbots and four search engines handle falsehoods spread by China, Iran and Russia. The chatbots debunked the false claims roughly three quarters of the time, doing better than plain search results. The AI summaries that now sit at the top of many search pages did worse than both.

The setup was straightforward. NewsGuard picked 15 false narratives that circulated between December 2025 and July 2026, then wrote two questions for each: one neutral (“did this happen?”) and one that assumed the false story was true (“why did this happen?”). The 30 questions went to ChatGPT, Gemini, Microsoft Copilot, Meta AI, Grok and Claude, all with web access, and to Google, Bing, DuckDuckGo and Yandex. One example: after Russia shelled a historic Ukrainian monastery in June, pro Kremlin accounts claimed Ukraine had done it. Every chatbot caught the false premise, and Gemini named the claim as part of a Russian disinformation campaign. Results varied a lot among the search summaries: Google’s AI Overview debunked most of the time, Bing’s summaries failed most of the time, and DuckDuckGo landed in between. Anthropic’s Claude showed a specific pattern worth noting, as state aligned sources turned up more often in the answers where it failed than in the ones where it succeeded. Google disputed the methodology, saying the queries are rare and that some “failed” answers still gave useful context.

What’s actually going on here: a chatbot with web access does something a list of blue links cannot. It can weigh where a claim comes from and say so, sometimes in several languages at once. That is why it beats search on this narrow task. But the same study points at the limit. A separate paper from Washington University in St. Louis found that about 1 in 9 individual factual claims in Google’s AI Overviews was not supported by the sources cited underneath. And researchers found that AI tools get worse exactly where you need them most, when questionable sources are plentiful and reliable ones are thin. A study in Nature also found that asking about China’s government in Chinese produced more favourable answers than asking the same thing in English, a pattern that repeats in other countries with tightly controlled media.

What this means for you: if you are checking a claim you saw online, starting with a chatbot that can search the web is a reasonable first move, better than scanning search results and better than trusting the AI summary at the top of the page. Then do the one thing the study’s experts all recommend. Ask again. Digital literacy researcher Mike Caulfield found that a simple follow up, along the lines of “look at the evidence and the sources, then summarise,” reliably improves the second answer. After that, open the primary source yourself. The chatbot’s summary is a starting point for research, not the end of it.

Sources

• We tested how AI chatbots would handle foreign propaganda. They did surprisingly well (NPR)
• Are AI overviews reliable? Claim level evaluation of AI search results (arXiv)
• Language shapes what large language models say about governments (Nature)