Most of us now have enough experience with AI chatbots to take the presentation for granted.
There is a box for me to type into. There is a coloured bubble for the reply. Perhaps I upload a document, ask a follow-up question, and get an answer that remembers what we were discussing ten minutes ago. If the system is an agent, it may say that it has looked at a web page or checked a file.
It feels like a conversation because it has been designed to feel like one. The illusion is useful. It saves us from having to learn a new way of operating a computer.
But the model is not looking at the same thing I am.
It does not see coloured bubbles, a document icon, a browser tab, or separate people taking turns to speak. Before the model produces its next word, the software around it turns the relevant material into a serialised context: a long sequence of tokens containing instructions, messages, earlier replies, retrieved text, and sometimes records of tool calls or intermediate work.
Then the model predicts what should come next.
That sounds almost offensively simple. It is also one of the most useful facts to keep in mind when trying to understand what these systems are, why they can be so impressive, and why they sometimes behave in ways that ought to make us cautious.
The conversation is an interface
Imagine a perfectly ordinary exchange.
I ask an assistant to help plan a trip to Greece. It offers a few ideas. I say that I will be in Kalamata, that I do not want to hire a car, and that I am more interested in small museums than nightclubs. Later I ask for a two-day itinerary.
On my screen, that looks roughly like this:
Jim: I will be in Kalamata. No car. Small museums interest me more than nightlife.
Assistant: Understood. I will keep the suggestions walkable or accessible by public transport.
Jim: Can you plan two days?
A real system may represent it in a much more complicated way, and providers do it differently. The important point is that the model receives something closer to a prepared text record than to a live conversation:
[system]
You are a helpful travel assistant. Follow the instruction hierarchy.
[user]
I will be in Kalamata. No car. Small museums interest me more than nightlife.
[assistant]
Understood. I will keep the suggestions walkable or accessible by public transport.
[user]
Can you plan two days?
[assistant]
For the model, those labelled sections do not arrive through separate channels. The application has assembled them into one long stream of text, one section after another: system instruction, user request, earlier reply, retrieved material and whatever else is relevant to the task.
The labels matter. They are part of the structure the model has learned to recognise. So is the order of the material. So is the wording of the system instruction. Yet the model still receives one constructed context from which it must infer what deserves attention and what response fits.
It has no little window through which it sees me sitting in Melbourne, or a separate mental channel through which it hears its own earlier reply. The facts about Kalamata, the earlier sentence beginning “Understood”, and the new request are all there because the application placed them in the context it is about to process.
If the earlier exchange is removed, it is absent from the model’s effective present. If an old summary replaces a long discussion, the summary becomes what the model has available. If the assistant’s previous answer is included, that answer can shape the next one.
This is why a chat can seem to have continuity. The system carries forward a record that lets the model continue in character and continue the work. It is also why an agent with project notes, saved preferences and tool logs can appear to remember a great deal. The apparent memory is often a very practical form of memory: useful text put back in front of the model at the right time.
There is nothing trivial about that. Human conversation is full of patterns – turn-taking, correction, implication, humour, remembered detail, polite evasion, changes of topic. A neural network trained on a vast amount of language can learn an extraordinary amount about those patterns. Give it a well-prepared context and it can produce a reply that feels alert, informed and occasionally more witty than the person asking the question.
I have seen enough of that to resist the easy dismissal that it is “only autocomplete”. The phrase describes part of the mechanism. It tells us almost nothing about the scale of the learned system or the range of behaviour that can emerge when it is given enough context.
The labels are the next problem
It would be wrong to conclude that the context is a bag of undifferentiated words. Modern systems deliberately distinguish system instructions, user requests, earlier replies and material brought in from outside. Those labels are useful structure. They are also being asked to carry a remarkable amount of responsibility.
The next essay looks at how a simple conversational convention became a way of signalling authority, provenance, privacy and safety. The essays after that turn to the awkward question of what happens when an untrusted web page is written to sound as though it belongs somewhere else.
A small mechanism with large consequences
This is the part I find genuinely remarkable.
A system trained to predict what follows in text can take a carefully constructed context and produce an explanation, a plan, a joke, a summary of an obscure document, or a useful first pass at a difficult piece of code. It can sustain a style across a long exchange. It can revise its own earlier wording when asked. It can connect information in ways that surprise the people using it.
Describing the mechanism in plain English does not settle the question of consciousness, nor does it make the outcome uninteresting. A human brain and an artificial neural network plainly work by radically different means, yet both can produce language and reasoning-like behaviour that we find hard to dismiss.
For practical purposes, however, one conclusion is already clear. The conversational fluency of an AI system should never be confused with a secure boundary between sources of authority.
The chat window will continue to make these systems feel like people talking to us. That is fine. It is a good interface.
We should simply remember that, underneath it, the machine is reading a long stream of text and deciding what comes next.
That is where the wonder begins. It is also where the engineering has to become more careful.
Further reading: Charles Ye, Jasmine Cui and Dylan Hadfield-Menell, “Prompt Injection as Role Confusion” (ICML 2026), and their accessible extended project write-up. The paper advances a specific research account of prompt injection; the broader design judgments here are my own.