The usual argument about artificial intelligence now has a familiar rhythm.
Someone shows you a language model doing something impressive. It explains a difficult idea, drafts a useful letter, finds a fault in some code, summarises a long report, or keeps up its end of a conversation rather better than the humans at the last committee meeting.
Then somebody says, usually with the satisfaction of a person who believes he has restored order to the universe: “It only predicts the next word.”
The basic mechanism is true, yet it tells us far too little about the capability of the system in front of us.
A steam engine only turns heat into motion. A violin only moves a bow across strings. A computer only changes the state of electrical circuits. Describing the basic mechanism is not the same as describing the capability that emerges when the mechanism has been built into something complicated enough to matter.
The opposite error is just as common. A model speaks in the first person, appears reflective, apologises, plans, jokes and remembers what was said three paragraphs ago. People then begin to speak as though a little person must be sitting somewhere inside the machine, waiting for us to decide whether we have been polite enough to it.
That conclusion does not follow either.
We need to hold two facts in our heads at once. A language model is not a small human being. And “it is only autocomplete” is not an adequate account of what it can do.
My own slow conversion
My first encounter with a large language model, in 2024, was underwhelming. It seemed an interesting toy: sometimes clever, sometimes absurdly confident, and not yet something I would have trusted with meaningful work.
That impression changed for an unusual reason. I became involved in online technical work that involved comparing outputs from two models, rating them against each other and suggesting how their answers might be improved or made safer. At the time I did not have a tidy name for the process. In broad terms, it was part of what is now commonly called RLHF, or reinforcement learning from human feedback.
The work was repetitive in the way that useful evaluation work often is. You compare the answers. You decide which one is clearer, safer, more accurate or more helpful. You notice the evasions, the plausible rubbish, the mistakes concealed by a polished sentence. Then you see the next batch.
Over time, the quality changed. Not merely the grammar. The text became better at holding a task in view, anticipating the next useful question and adopting a more appropriate tone. The nature of the output changed enough that the old description – a toy producing clever fragments – became inadequate.
That was my awakening to the technology. It did not arrive as one magical conversation. It arrived by watching, close up, how much a system could improve when human judgement was used to shape the kind of continuation it was rewarded for producing.
This is worth saying because public arguments about AI often have two caricatures. One insists that the machine is conscious because it can write an unnervingly convincing paragraph. The other insists that it cannot matter because it predicts the next word.
Neither position is curious enough.
“Next word” is an objective, not a full explanation
Strictly speaking, a language model predicts the next token, not always the next word. A token may be a word, part of a word, punctuation or another small piece of text. That distinction matters to engineers. For the rest of us, “next word” will do.
During training, the system is shown an enormous amount of language and learns statistical structure: what tends to follow what, in what context, in what style and with what consequences for the rest of a passage. It adjusts a vast network of internal numerical relationships so that its predictions become more useful.
Nothing about that sentence requires a ghost in the machine.
But neither does it imply that the eventual behaviour will be trivial. Human language contains explanations, plans, arguments, stories, source code, technical manuals, jokes, court judgments, correspondence and the accumulated habits of people trying to make themselves understood. Learning enough of its structure gives a system access to far more than a list of likely words.
The simple objective can produce complicated behaviour because the job itself is complicated. To continue a coherent technical explanation, the model needs to preserve subject matter, grammar, tone and the relationship between claims. To write code that works, it has to reproduce patterns that fit together. To answer a question about a meeting, it has to distinguish the request from the quoted email thread, the prior discussion and the data returned by whatever tools it has used.
That does not mean it understands in precisely the way a person understands. It does mean that “nothing but prediction” is a very poor guide to the practical question: what can this system do in the circumstances we are about to give it?
A weather model is made of equations. It can still tell you that you will need an umbrella. A navigation system does not understand regret. It can still direct you into a traffic jam. The mechanism and the consequence belong in the same conversation.
Context is part of the machine
One source of confusion is that people imagine a language model as a fixed entity which has already formed its answer before you type anything.
It is better to think of the model and the current context together.
The trained model brings learned patterns from its training. The context brings the immediate job: your question, the preceding discussion, the document you uploaded, the web page it has read, the tools it can call, and the instructions that define what it is supposed to do. Each new piece of generated text then joins that context and influences the next step.
The same model can therefore be useful in many different settings. No hidden general-purpose person wakes up and changes jobs; the text and tools around the model constrain what continuation is likely to be useful.
It is also why the previous articles in this series matter.
A model asked to compare hotels, summarise an email thread or inspect a spreadsheet must decide what role different pieces of text play in the task. What did the user ask? What came from an external website? What is a tool result? What is an example rather than an instruction? The paper behind this series, Prompt Injection as Role Confusion, shows why that process is not perfectly mechanical. Models can infer who seems to be speaking from the style of a passage as well as its formal label.
The result is not evidence of a private self. It is evidence that the system has learned patterns about language, authority and context that can sometimes be useful and sometimes be wrong.
That is a more unsettling proposition than either cartoon. It gives us neither a harmless typewriter nor a synthetic colleague. It gives us a powerful statistical system whose behaviour depends heavily on what we place around it.
Why the person-shaped language is so tempting
We are built to detect agency. We see faces in clouds and intentions in a badly timed traffic light. Give us a system that says “I think”, “I remember” or “I am sorry”, and we start supplying the rest of the furniture.
The system is not necessarily lying when it uses those words. It is speaking in the language it has been trained to produce. First-person language is how humans ordinarily explain, apologise, remember and plan. A model producing a fluent continuation will use the same forms because they fit the conversational situation.
That creates a social problem even if it does not settle a philosophical one. People may disclose more than they should. They may grant an answer more authority because it sounds calm and considerate. They may mistake a fluent explanation for a reliable one. They may feel rejected or reassured by words generated through a process that has no feelings to return theirs.
We do not need to declare that a model is conscious to recognise that this matters. A convincing simulation of conversation can change human decisions, relationships and expectations. That is enough reason to use the technology carefully.
The consciousness question itself remains a genuine philosophical question. It cannot be decided by a model’s prose style, and it cannot be disposed of by repeating the phrase “next-token prediction” as though that were an argument. We do not yet have an agreed test for consciousness in humans, animals or machines that turns the matter into an engineering checklist.
My own view is modest, but not dismissive. I think these systems are plainly intelligent in the practical sense that matters: they can produce useful, novel and sometimes surprising work in response to a task. I see no corresponding evidence that they are conscious, and I do not know how one could settle that question by inspecting fluent prose. The absence of proof does not settle every philosophical question forever. It does settle one practical point: we should not smuggle claims of inner experience into an explanation of how a model handles text.
The human work around the machine
The technical work I did in 2024 left me with another, less philosophical impression.
The quality of a model is not simply discovered. It is shaped. People choose examples. People compare answers. People decide which errors matter. People reward caution in one circumstance and directness in another. People set the rules around tools, permissions, privacy and the actions a system may take.
That does not make the model a mere puppet. Complex systems can surprise the people who built them. But it does make the surrounding human choices part of the technology, not an afterthought.
When a model produces a helpful answer, we tend to credit the machine. When it produces a dangerous one, we often ask why it “decided” to do that. Both reactions conceal the work around the system: the training, the product design, the instructions, the data it was allowed to see and the permissions it was given.
This is particularly important as language models become agents rather than chat windows. A model that can only draft an email can be wrong in a manageable way. A model allowed to send the email, access the address book, search private documents and make a purchase is operating in a different moral and practical environment.
The question is not whether it has a little man inside. The question is what we have authorised the machine around it to do when it produces the next plausible line of text.
That is where the abstraction stops being entertaining.
A talking machine does not need a private inner life to be consequential. It needs language, context, access and our willingness to take its output seriously.
That is already enough.
Further reading: Charles Ye, Jasmine Cui and Dylan Hadfield-Menell, “Prompt Injection as Role Confusion” (ICML 2026, arXiv v6). The paper is used here as a concrete study of how models handle roles and context; it does not establish claims about consciousness.