A modern AI interface makes an extravagant promise without ever quite saying so.
It gives us a conversation. There are coloured bubbles. There are named speakers. A file appears to have been uploaded, a web page to have been read, a tool to have returned a result. The model replies in fluent English, remembers the thread, offers a plan, acknowledges a correction and sometimes apologises with more grace than a telecommunications company.
The arrangement invites us to believe that the important boundaries are already there.
The user is the user. The assistant is the assistant. A web page is external information. A plan is a plan. A private note is private. A safety instruction is higher than an email from a stranger. The software surely knows the difference.
This series began with the inconvenient fact that, inside a language model, the picture is less tidy. The system receives a serialised context: text from several sources, marked and arranged in ways intended to give it structure. It then predicts what should follow.
That does not mean the model has no structure. It has role labels, training, position, patterns learned from a great deal of language and rules imposed by the application around it. The point is more modest and more important. The boundaries we see in the interface are not necessarily hard boundaries inside the model.
A passage can look and sound like an instruction even when the application has labelled it as data. A fabricated working note can receive more trust than it deserves. A plan can look like a commitment while remaining a paragraph in a context window. These are different examples of the same failure: text has acquired an authority the surrounding system did not truly establish.
None of this makes language models mystical. It makes them what they are: powerful systems for continuing and interpreting language, working in environments where language is being asked to carry an extraordinary amount of authority.
The limits of a better prompt
Clear instructions still improve behaviour. They should be written well, tested and kept in their proper place. But they do not change the basic fact that the model must interpret the words we supply. In a consequential system, “usually gets the hierarchy right” is not an architecture.
The practical response is not to abandon language models. It is to ensure that authority is enforced somewhere more reliable than the next plausible sentence.
The agent is not the whole application
This point becomes more pressing as chat tools become agents.
A chatbot that gives a bad answer may waste ten minutes of your time or, at worst, persuade you of something foolish. That is not nothing, but it is a limited class of harm.
An agent may read a private inbox, search a file store, retrieve information from the web, create a document, submit a form, call an external service, change a record or send a message. The language model is one component in that chain. It should not become the sole interpreter of what it is allowed to do.
A well-designed agent can be useful without being omnipotent.
It can read an invoice and prepare a proposed payment without being able to release the money. It can summarise a job application without becoming the final selector. It can draft an email without being able to send it unreviewed. It can inspect a codebase and propose a change without having credentials that allow it to deploy to production.
These are not half-measures. They are the difference between a helpful assistant and an unaccountable actor.
We need to stop measuring intelligence by how completely a product removes the human from the loop. In a consequential setting, the useful question is whether the human remains at the point where judgement, responsibility and authority are actually required.
Provenance is a practical virtue
One word sits behind much of this series: provenance.
Where did this text come from? Is it a user request, a policy, a retrieved document, a tool result, an earlier model response, a plan or a human approval? What authority does that source have? What is it allowed to influence? Can the system show a person the answer to those questions later?
Human organisations run on provenance more than we tend to notice.
A signature on a contract means something because we know whose signature it is, what document it belongs to, when it was made and what process gave it force. A medical result is useful because there is a chain from specimen to laboratory to clinician. A financial record can be audited because its origin and subsequent changes are not merely matters of tone.
Language-model systems will need the same discipline. It is not enough for text to be present in the context. The application needs a way to preserve and enforce what kind of text it is.
That may mean separate credentials for separate operations. It may mean keeping sensitive data out of broad model context. It may mean signed or structured approvals, restricted tool interfaces, read-only modes, isolated environments and logs that identify what the agent read before it acted.
Some of this will feel unfashionably ordinary beside a polished demonstration of an agent “handling everything”. Ordinary is not a criticism. Most of the systems we trust most are full of rather boring controls that have earned their place by preventing expensive mistakes.
The role of human judgement
It would be a mistake to answer this problem by declaring that humans must personally approve every tiny action forever.
That would make agents too slow for many of the jobs where they are genuinely useful. Nobody needs a board meeting before an agent sorts a folder, renames photographs or drafts an internal meeting summary.
The right question is proportionality.
What is the agent permitted to read? What is it permitted to change? What can leave the system? What cannot be undone? Who is answerable if it gets the interpretation wrong?
The more consequential the answer, the more we should prefer limited permissions, independent checks, visible proposed actions and the capacity to review or reverse what happened.
Trust in a model should be proportionate to the job and the controls around it. That is what adult trust looks like.
We do not trust a good employee by giving them every password in the company on their first day. We trust them by giving them an appropriate job, appropriate access, support, supervision and a clear route to ask when something is unclear. A system that cannot ask, cannot be checked and cannot be stopped is not more trustworthy because it acts confidently.
What this series is not saying
It is worth being clear about the conclusions this argument does not support.
It does not say that prompt injection makes AI useless. Models remain extremely useful for writing, summarising, searching, coding, analysing and assisting. Practical safeguards can reduce risk substantially even if no language-only defence is perfect.
It does not say that a fluent model has a hidden human mind, or that it has none. The question of consciousness is not settled by a chat window, a first-person pronoun or the phrase “next-token prediction”. It is an important philosophical question and a poor substitute for system design.
It does not say that every persuasive web page is an attack. A separate, deliberately deferred research note explores the still-unproven question of whether external prose can quietly influence later recommendations. That is not established evidence of an industry already manipulating AI agents at scale.
And it does not say that responsibility belongs to the model when a system fails. The responsibility remains with the people and organisations who chose the model, designed the product, connected the tools, set the permissions and decided what safeguards were worth omitting in the name of frictionless automation.
That last point may be the most useful one.
We are building social systems, not only technical ones
When an AI agent acts for someone, it does not merely process data. It enters a field of relationships: customer and supplier, applicant and employer, citizen and public institution, patient and practitioner, company and employee, family member and caregiver.
In each relationship, authority has a source. Consent has a meaning. Privacy has a boundary. A mistake has a cost borne by somebody in particular.
No arrangement of chat roles can relieve us of the job of deciding those things.
The current role labels – system, user, assistant, tool and their relatives – have done remarkable work. They helped turn raw text prediction into useful conversation and tool use. But they have gradually been asked to carry identity, hierarchy, privacy, planning, safety, provenance and control. That is a great deal to rest on a small set of textual conventions.
The next stage of AI engineering should not be another round of cleverer wording alone. It should be a clearer division between the model’s flexible work with language and the surrounding systems that establish permission, responsibility and consequence.
That is the point on which the whole dossier rests. Models can interpret. Tools can enforce. People can decide where authority belongs.
That is the boundary we need to build.
Further reading: Charles Ye, Jasmine Cui and Dylan Hadfield-Menell’s “Prompt Injection as Role Confusion” provides the central research frame for this series. NIST’s AI Risk Management Framework and Generative AI Profile provide voluntary guidance for managing AI risks through design, development, deployment and evaluation.