Readers Rated AI Short Stories Higher Than Human Ones, Right Up Until They Were Told a Machine Wrote Them
In three experiments with more than 2,500 people, readers did no better than chance at spotting ChatGPT fiction. The AI stories scored higher on quality and immersion, and the label alone was enough to drag the scores down.
A new study in the journal Judgment and Decision Making put more than 2,500 people in front of short stories and asked them to spot the machine. They could not. Across three experiments, participants performed no better than a coin flip, and the AI-written stories were rated more highly than the human ones, at least while readers believed a person had written them.
Researchers Sydney Sears and Deena Skolnick Weisberg ran the first experiment with 1,682 participants. Each read one story of about 1,000 words. Three of the six came from established literary magazines and short story collections. The other three were generated with ChatGPT, prompted to match the theme, style, and narrative perspective of the human originals. Half the readers were told a human wrote the piece, half were told ChatGPT did. That label was accurate for only half of each group, which is what makes the design useful: it separates what the text does from what the reader expects.
On a scale from minus 3 to plus 3, the AI stories scored 1.54 for perceived quality against 0.97 for the human ones, and 1.42 against 1.00 for immersion. The label mattered independently. Whoever had actually written it, readers gave higher marks when told the author was human. Readers who liked AI in general rated everything higher and went higher still when told it was ChatGPT. Among AI-sceptical readers, the effect ran the other way. In two further experiments with 905 people, participants read one human and one AI story side by side and had to say which was which. Even with a direct comparison, they were at chance.
Before anyone concludes that ChatGPT writes better fiction than novelists, the authors offer a deflating explanation and it is probably the right one. AI prose tends to be smoother, easier to read, and more emotionally upbeat, and people reliably prefer things that are easy to process. Serious literary fiction often does the opposite on purpose, making you work for the meaning. A story can be high quality and not very pleasant to get through. The 1,000 word format also flatters the machine, since holding a story together for a page is a different problem from holding one together for 300. Experience with AI tools correlated with spotting the fakes. Experience with fiction did not.
What this means for you: the practical takeaway is not that AI writes well, it is that you cannot tell by reading, so any confidence you have in your own detector is misplaced. If you commission or publish writing, this is an argument for judging work on what it does rather than trying to sniff out its origin, and for asking about disclosure directly instead of guessing. If you write, the finding is more encouraging than it looks: the qualities that made AI stories win here, smoothness and easy emotional payoff, are exactly the qualities that make forgettable prose. And if you are simply a reader, it is worth noticing how much your enjoyment moved based on a label rather than the words. That says something about all of us. All data and materials are public on the Open Science Framework, so you can read the stories and judge for yourself.
Sources
xAI's New Image Model Lands Second in the Arena Rankings, and Brings Photoshop-Style Editing to Grok
Imagine Image 2.0 scores just behind OpenAI's GPT-Image-2 on both public leaderboards. The more interesting part is the editing tools, which point at where image AI is actually heading.
