There is a reasonable-sounding argument that AI should make people more creative. It has ingested more written material than any human could read in a hundred lifetimes, it can generate a hundred variations before you finish your coffee, and it never gets tired of a brief. The argument has a flaw, and the flaw is structural rather than incidental: a language model is trained to predict the most probable next token given everything that came before. Creativity, in any sense worth having, is the opposite operation. It is the deliberate production of something improbable that turns out, on reflection, to be worth more than the probable thing.
What "the way it's supposed to be done" actually optimises for
When you ask a model to write something, edit something, or solve something "properly", you are asking it to do the thing it is best at: converge on the response most consistent with everything similar it has seen before. A team studying this recently traced the effect to something more specific than an algorithmic quirk. They found a typicality bias baked into the human preference data used to fine-tune these models: the people rating model outputs during training systematically rewarded familiar phrasing over unfamiliar phrasing, a well-documented tendency in cognitive psychology applied at scale, across millions of ranking decisions. The model learns not just what is likely, but what raters found comfortable, and comfortable and familiar are close to the same thing. The result, replicated across model families, is what researchers call mode collapse: the model's outputs cluster tightly around a narrow set of "safe" responses even when the prompt space is enormous.
The clustering is measurable and it is not subtle. One study generating 20,000 stories across four major model families found that 88.3 percent of the stories shared one of just eleven recurring core words (character names, settings, professions) regardless of which model wrote them. A separate study comparing large language models directly against humans on standardised creativity tasks (the kind cognitive psychologists have used for decades to measure divergent thinking) found that LLM responses resembled each other far more than human responses resembled each other, even after controlling for response format. It is not that any one model is boring. It is that all of them are boring in the same way, because they were all optimised against the same instinct: reward the response that sounds like it belongs.
A third study, testing fourteen widely used models against validated creativity benchmarks, found that on the standard Alternative Uses Task, models outperformed the average human. But only 0.28 percent of model-generated responses reached the top decile of human creative output. The models are reliably competent and almost never remarkable, and reliably competent is exactly what "the way it's supposed to be done" produces when the person doing the asking wants the safe, well-formed, unsurprising version of the task done quickly.
Three kinds of new, and where the machine stops
The philosopher and AI researcher Margaret Boden gave this problem a useful shape decades before anyone had a language model to test it against. She distinguished three kinds of creativity. Combinational creativity takes two familiar ideas and puts them together in a new pairing; most metaphor works this way. Exploratory creativity takes a given conceptual space (a genre, a method, a set of rules) and works out what else is possible inside it without changing the rules themselves. Transformational creativity changes the rules of the space, so that ideas which were previously not just unlikely but literally unthinkable within the old framework become available.
The distinction matters here because it maps almost exactly onto where the research says these systems succeed and fail. Researchers evaluating text-to-image and language models against Boden's framework have observed the same pattern repeatedly: models handle combinational creativity well, handle exploratory creativity reasonably, and struggle badly with transformational creativity, because transformational creativity requires abandoning the very structure the model was trained to reproduce. A model trained to predict the next most probable token, conditioned on a lifetime of examples of "how this genre works", is architecturally suited to explore that genre's interior and unsuited to questioning whether the genre's boundary should exist at all. It cannot un-learn the shape of the space in order to ask if a different shape would be better, because the shape of the space is the only thing it has to work with.
This is the part of the AI-creativity conversation that gets skipped. It is not that models cannot produce novel combinations. They can, quickly and cheaply. It is that the discovery people actually mean when they talk about creative breakthroughs, the moment someone realises the rule itself was optional, sits in the one category these systems are least equipped to reach, for the same reason they are good at everything else: they were built to model what already exists.
The exploration you have to supply yourself
None of this is fixed. A recent prompting technique called Verbalized Sampling asks the model to output a distribution of responses with their associated probabilities instead of a single best answer, in effect asking it to admit what the unlikely options look like rather than silently discarding them. The technique measurably widens output diversity across creative writing, dialogue, and open-ended tasks, without a loss in accuracy. It works because it forces the model to surface the tail of its own distribution instead of collapsing to the mode by default.
But notice what the technique actually does. It does not make the model transformational. It makes the model show you more of its combinational and exploratory range, and it does this only because a person asked for the distribution instead of the answer. The judgment about which of those unlikely branches is worth pursuing, which deviation is interesting rather than merely strange, still has to happen somewhere, and it does not happen inside the sampling. Boden made a version of this point about human creativity long before anyone needed it applied to models: generating an unusual combination is not the hard part. The hard part is evaluating whether the combination is any good, and evaluation depends on a structure of values the generator does not itself possess.
This is the same shape of argument this site has made about expertise: the tool amplifies what you bring to it and cannot substitute for the part you don't. The creative case is structurally identical, with one twist that makes it sharper. In the expertise case, the danger is skipping the struggle that builds judgment. In the creative case, the danger is that the "correct" way of doing something is precisely the well-lit path the model was built to walk, and the interesting result was never going to be found by walking it faster.
Doing it wrong on purpose
Put a person and a model in front of the same brief and ask for "the right way to do this", and you will get, from either of them, the version that looks most like what has worked before. That is not a failure specific to AI. It is what "the right way" means: the accumulated, averaged, previously-validated method. The discoveries that later get called creative breakthroughs are, almost definitionally, the ones that happened when someone deviated from that average and the deviation turned out to matter. Penicillin was a contaminated petri dish someone chose to look at rather than discard. The Post-it note came out of an adhesive that failed at the one job it was designed for. Neither came from doing the procedure correctly.
What a model changes is the cost of generating the deviation. It has never been cheaper to produce a hundred variations on a theme. What it does not change, and structurally cannot change on its own, is the judgment required to notice which variation is the contaminated petri dish and which is just contamination. That judgment is not a matter of more data or better sampling. It is a matter of values, of caring about a particular kind of surprise, of having enough command of the "correct" version to recognise the moment a wrong one is actually right. The direction, which rules are worth breaking and why, is still a human task. The machine can widen the field of variations enormously. It cannot tell you, from inside its own training, which unlikely thing you should have wanted all along.