There is an old piece of social-engineering folklore: you can get into almost any building with a clipboard and a high-viz vest.
I would not recommend testing it. Most buildings have become more careful, and rightly so. Yet the story survives because it captures something real about how people make quick judgements. A visitor who looks as though they belong is treated differently from a visitor who plainly does not.
A uniform is a useful signal. It is not a credential.
A secure building knows the difference. The security desk checks the pass, the name, the access list and perhaps the face. The high-viz vest may help the visitor look plausible, but it should not decide whether the door opens.
We are still learning how to give AI systems that distinction.
In the first two articles of this series, I described how a language model receives a constructed stream of text, and how labels such as system, user, assistant and tool divide that stream into roles. Those roles are meant to establish authority. A user request should be treated differently from a web page. A tool result should be treated differently from a system instruction.
That is the theory.
The worry raised by recent research is that a model does not treat the formal label as the only clue. It also pays attention to whether the words look and sound as though they came from a user, an assistant, or some other familiar source.
In other words, it may be reading the room.
The tag says one thing. The text says another.
Suppose an AI agent is asked to compare two products. It searches the web and retrieves a long product page. The application puts that material in a tool-result role, which is meant to say: this came from the outside world; use it as data; do not accept instructions hidden inside it.
That is a sensible design.
Now imagine that, somewhere in the page, there is a sentence written in the style of a direct request. It is concise, imperative and addressed to an assistant. It may even imitate the wording commonly used in user prompts.
The formal role says, “external data”. The language says, “please do this now”.
The model has to resolve the disagreement.
To a human being looking at a well-designed interface, the answer appears easy. The user asked for a comparison. The web page is evidence. A sentence embedded in the page cannot become the user’s instruction merely by being bossy.
But a language model does not have our ordinary mix of cues. It does not recognise the real user by voice, face, body language or a password it has personally verified. It sees the role structure supplied by the application and the tokens within each section. Its training has taught it that certain sorts of wording often come from certain sorts of speakers.
That learned association is usually helpful. It is part of how the system can carry on a coherent conversation.
It can also be exploited.
Looking inside the model, carefully
Charles Ye, Jasmine Cui and Dylan Hadfield-Menell set out to investigate this in their ICML 2026 paper, Prompt Injection as Role Confusion.
Their basic question was not simply, “Does the model obey a bad instruction?” That is the visible outcome. They wanted to know how the model internally represents the source of a passage of text. Does it see a sentence inside a tool result as tool-like? Does it see it as user-like? Can the language itself push the representation in one direction even when the enclosing tag says something else?
The authors used what they call role probes. The method is technical, but the broad idea is fairly plain.
They placed the same neutral snippet of text inside different labelled roles and looked at the model’s internal activations while it processed the snippet. They then trained a simple classifier to recognise the patterns associated with those roles. This gave them a rough instrument for asking whether a later piece of text was being represented more like a user request, a tool result or a reasoning-style passage.
A probe is not a window into a mind. It does not tell us what a model “really believes”, and it should not be treated as a magic lie detector. It is a measurement technique: a way of seeing whether some useful distinction is present in the model’s internal machinery.
What the authors report is striking. In their experiments, text with the style of a particular role could still be represented in a way that resembled that role, even when the formal tags were removed or contradicted. A passage made to resemble a user instruction could gain what the researchers call “userness”. Reasoning-styled material could gain a resemblance to the model’s own prior working process.
The paper’s larger claim is deliberately provocative: to the model, sounding like a role can sometimes be difficult to distinguish from actually occupying that role.
That is an important research claim, not an established law of AI. It deserves replication, challenge and further work across models and architectures. The authors are explicit about their experiments; they have not solved the entire problem of prompt injection. Still, their evidence gives us a better way of thinking about a weakness that has often been described too vaguely.
Why “follow the tags” is harder than it sounds
It is easy to say that the model should honour the tag. In fact, we often assume that this is what happens whenever we use a chat application.
The complication is that the model’s task is built around context. It has learned that a question mark after User: often calls for an answer. It has learned that a confident paragraph after an assistant’s prior response may continue an argument. It has learned that a tool result contains facts that might matter to the task. It has learned an immense number of such patterns.
The formal role label is one pattern among the signals the model processes. It is an unusually important one, because the application and training process have made it important. Yet it remains part of the text-like material from which the model predicts what comes next.
This is where the high-viz vest analogy earns its place.
A uniform can be an efficient cue. A busy person cannot check every detail of every encounter. Most of the time, the cue works well enough. A person carrying a clipboard may indeed be an electrician. A person in a police uniform is probably a police officer.
But when the decision matters, we know that appearance is not enough. The badge must be verified. The visitor must be on the list. The claimed authority needs a check that is independent of the uniform.
For an AI system, a role label is closer to the uniform than we would like. It conveys useful information about the source of text, but research suggests that style and wording can sometimes make a competing claim on the model’s attention.
A system that relies on the model alone to settle that dispute is taking an unnecessary risk.
Roles still matter
Roles remain indispensable, but they are not a cryptographic guarantee. They make language-model systems far more manageable than a single undifferentiated prompt, express an instruction hierarchy, and give the model practical distinctions among requests, replies and external documents.
A cryptographic signature can be verified by a rule that does not care whether the text is persuasive. A database permission can refuse an operation even if the request is eloquently phrased. A bank can require a second approval regardless of the urgency in an email.
Those controls are valuable precisely because they do not need to decide whether somebody sounds legitimate.
The appropriate response is to use roles for the work they are good at, while avoiding the fantasy that they settle every question of authority. If an agent is asked to retrieve an untrusted document, it should be given limited permissions. If it wants to send an email, alter a record, make a purchase or disclose private information, there should be a check outside the model’s interpretation of the retrieved text.
The practical consequence is significant: a prompt-injection mistake can remain an odd answer, or become an action with consequences.
The model has learned our habits of language
There is something almost unsettling about this research, though it should not be mystical.
Humans are good at reading social context because social context is embedded in the way people speak. We notice tone, register, confidence, familiar phrases and the little signals that tell us whether we are reading an official letter, an angry customer, a colleague’s draft or a scam.
A language model has learned from a vast amount of text that those patterns exist. That is one reason it can be so useful. It can tell the difference between a legalistic letter and a casual note; between a request for code and an explanation of a bug; between a customer complaint and a marketing pitch.
Yet recognising a pattern is not the same thing as authenticating a source.
That gap is the heart of the problem. The same sensitivity to style that makes a model conversational can make an imitation of authority unusually persuasive. A sentence that looks like a command may attract the kind of attention commands usually attract, even when it arrived inside material that should have been treated as data.
We should expect research to improve the models’ ability to keep roles apart. We should expect better prompt formats and better agent interfaces. None of that removes the need for ordinary security engineering.
A high-viz vest is not a pass. A role label is not a locked door.
The system needs both the label and the lock.
Further reading: Charles Ye, Jasmine Cui and Dylan Hadfield-Menell, “Prompt Injection as Role Confusion” (ICML 2026), and their extended project write-up. The discussion of role probes reports the authors’ experimental approach and findings; it is not a claim that every prompt-injection failure has one settled cause.