A new study just handed the literary world an uncomfortable twist: readers can’t reliably tell the difference between a short story written by a human and one written by ChatGPT. Worse for the humans — when people did rate the stories, they often preferred the AI’s version. But the moment you tell them a computer wrote it, they turn on it instantly.
Welcome to the strange psychology of “bot or not.”
The Setup
Researchers Sydney Sears and Deena Skolnick Weisberg at Villanova University ran three linked experiments — published August 5 in the journal Judgment and Decision Making — to settle a simple but loaded question: can ordinary readers spot AI fiction when it’s sitting right in front of them?
They took three short stories that had actually been published in literary magazines and collections, then had ChatGPT-4 write matching stories on the same themes — grief and koi fish, a man watching a woman dance, an air-raid-era government worker questioning his purpose. Over 3,000 adult participants, recruited to mirror the U.S. population, were brought in to read and judge.
Round One: The Reveal Backfires
In the first experiment, each participant read just one story — either the human original or the AI version — and was told (sometimes truthfully, sometimes not) who supposedly wrote it.
The results were almost mischievous. Stories actually written by ChatGPT were rated as more absorbing and higher quality than the human-written ones. But regardless of which story someone actually read, simply being told “a human wrote this” boosted their rating — and being told “AI wrote this” dragged it down.
In other words: people liked the AI’s writing better, right up until they found out it was the AI’s writing.
Round Two and Three: The Guessing Game
The next two experiments took the training wheels off. This time, everyone read both a human story and its AI twin, back to back, and had to guess which was which.
They basically couldn’t. In the second experiment, participants correctly identified the human-written story only 39% of the time — meaningfully worse than a coin flip. In the third, they landed almost exactly at chance, roughly a 50/50 guess.
Even more telling: the traits people leaned on to make their decision — how “enjoyable” a story felt, how easy the language was to follow — were the very traits that led them astray. The stories that felt smoothest and most pleasant to read were disproportionately the AI ones, and readers wrongly took that ease as a signature of human authorship.
Who Actually Could Tell?
Not literature buffs. Self-reported expertise in fiction — voracious readers, English majors, people who write for a living — had zero correlation with getting the right answer.
The one thing that helped, modestly, was familiarity with AI itself. People who used AI tools often and understood how they worked did slightly better at sniffing out the synthetic story — suggesting the skill of detection may have more to do with recognizing a machine’s fingerprints than with literary taste.
Why This Matters Beyond the Lab
The researchers point to a phenomenon called “algorithm aversion” — a well-documented human tendency to distrust machine output even when it’s objectively just as good, or better. It shows up in AI-written advice, AI-generated art, and now, fiction.
Their theory on why AI stories read so well in the first place is almost unsettling: large language models are, in effect, averaging together the patterns of millions of stories they’ve absorbed — smoothing out the rough, idiosyncratic edges that make individual human writing occasionally clumsy but also distinct. It’s the literary equivalent of the well-known finding that a face generated by blending many real faces together is rated as more attractive than any single real face. Familiarity and fluency, it turns out, read as quality — right up until the label says “AI.”
The Takeaway
We say we want art with a human soul behind it. But put to the test, blind and unlabeled, we often can’t find that soul at all — and we quietly prefer the machine’s version anyway. The bias against AI-made stories isn’t really about the stories. It’s about the label.
The study, “Bot or not: Can people tell the difference between stories written by a human or by an AI system?” by Sydney Sears and Deena Skolnick Weisberg, was published August 5, 2026, in Judgment and Decision Making.
