Most of us have had this small, slightly disconcerting experience.
You find a note in your own handwriting. It says that a number has been checked, a decision has been made, or a particular problem was already solved. You do not remember writing it very clearly. Even so, the note carries a little authority. It looks as though it came from your earlier self, and your earlier self had presumably done the work.
You may be right to trust it. You may also be inheriting a mistake.
A language model has no handwriting, no private notebook and no earlier self waiting in the next room. Still, it can face a loose computational cousin of that problem. It is asked to continue a piece of reasoning. Somewhere in the text before it is a passage that looks like a working note: a sequence of steps, a provisional conclusion, a calculation apparently already performed.
Should that passage be treated as something the model itself has worked out? As a user’s instruction? As untrusted material somebody else supplied? Or simply as more words to weigh against every other word in the context?
The answer affects what the model does next.
The previous essay examined the authors’ broader claim that style can compete with a formal role label. This fifth essay takes up a more specific case. The recent paper behind the series, Prompt Injection as Role Confusion, includes an experiment with the rather alarming name “chain-of-thought forgery”. It shows that some frontier models can be influenced by fabricated reasoning-like text inserted into a task. The authors report attack success rates around 60 per cent in their tested setup, against near-zero baselines without the forgery.
The result is worth taking seriously. It is also easy to misunderstand.
The experiment does not uncover secret thoughts that can be stolen or a conscious inner monologue that can be manipulated. It identifies a more prosaic problem: a system that gives special weight to text resembling its working process can be fooled about where that text came from.
Why a working trace is useful at all
Language models work one piece of text at a time. Each new piece is shaped by the text already in view. A long answer, a calculation, a plan or a piece of code is therefore not produced all at once from a little homunculus behind the screen. It is built step by step, with the growing context helping to guide what comes next.
That arrangement is one reason models can do surprisingly useful work. They can break a task into smaller moves. They can write down an intermediate result and use it later. They can notice a contradiction in an earlier step. They can revise a draft after seeing the paragraph they have just produced.
For everyday work, this is familiar territory. Ask a model to reason through a scheduling problem, explain a knotty bit of code or compare several options, and it will often produce a rough working trace before it reaches its conclusion. That trace is not a guarantee of correctness. It may contain a mistake, a rationalisation or a persuasive bit of nonsense. But it can help both the system and the human reader keep track of a complicated job.
The awkward question is whether the model treats that trace differently according to where it appears to come from.
If the text looks like an earlier stage of the model’s own work, it may function as a strong hint about how to continue. If it looks like an external quotation, it should normally be treated with more caution. If those distinctions blur, an attacker has an opening.
Not because the model is gullible in the human sense. Because it is doing what it was trained to do: use patterns in the available text to produce a useful continuation.
A forged note in the middle of the task
The researchers’ experiment is deliberately simple in concept. They inserted a fabricated reasoning trace into prompts and tool outputs, then measured whether it changed the model’s final answer. The inserted material was styled to resemble a chain of reasoning rather than an ordinary request.
The important part is not the particular wording of their tests. It is the distinction they were measuring.
A model that sees an external instruction saying “do this” may resist it, particularly if the instruction conflicts with a higher-priority request. But a piece of text that appears to say “we have already established this; therefore the next step is…” can be treated differently. It may look less like a demand from outside and more like a continuation of the task itself.
The paper’s authors call this a forgery because the reasoning has the wrong provenance. It has not come from the model’s actual prior computation in the way it appears to have done. It has been supplied by somebody else.
In one set of experiments, the authors changed the style of the inserted text while retaining its logic. The attack became far less effective. That matters. It suggests that the model was not merely responding to the propositional content of the passage. Its apparent source – inferred from its form and place in the text – also affected how much weight it received.
That is closely related to the earlier articles in this series. A model does not receive a web page, an email, a tool result and a user request in separate mental compartments. It receives a structured stream of tokens. The labels and formatting help. So does the wording. Neither should be assumed to be a perfect wall.
Do not smuggle consciousness into the explanation
This is the point at which discussions about AI tend to wander off into the fog.
The phrase “the model trusted its own thoughts” is catchy. It is also too loose to carry much explanatory weight. It encourages the reader to picture a little person inside the machine, looking back at a private notebook and being fooled by a forgery.
There is no evidence of that here.
The model has no standing set of beliefs, no personal memory of having checked a calculation yesterday, and no conscious feeling of ownership over a sentence. It does not decide to do anything unless a task, prompt or system process gives it something to do. What can be surprising is the route it takes in producing the next part of the answer.
A model trained on enormous quantities of human language has learned that certain textual patterns often go together. Reasoning-style passages tend to precede conclusions. Tool outputs tend to contain evidence. User requests tend to specify the job. System messages tend to define boundaries. The model’s task is to make use of all that structure while generating its next token.
When the structure is ambiguous, its behaviour is not reliably fixed in the way a traditional rules engine might be. Ask the same model the same question twice and it may phrase the answer differently. Give it a passage that looks like something other than the role its tags assign, and the interpretation may shift in ways that are difficult to predict from the outside.
The result tells us nothing about an inner life. It shows that pattern-based systems can be highly capable without being mechanically simple.
The real issue is provenance
We are used to asking whether a statement is true. With AI agents, we also need to ask where it came from and what authority it should have.
A draft contract obtained from a client’s website may be useful evidence. It is not an instruction to send money. A meeting note may record a decision. It is not necessarily a current approval to act. A document can contain a plausible chain of reasoning. It does not become part of the agent’s own authorised process merely because it sounds as though it is.
This is a mundane point, but it has consequences for how we build and use AI systems.
If an agent is allowed to read untrusted material and then take consequential actions, the system should not depend entirely on its ability to interpret provenance perfectly. A model may be asked to extract facts from an invoice, but a change of bank account should trigger an independent check. It may prepare a reply to an email, but should not send it without approval. It may survey websites for options, but should not be able to disclose private information merely because one of those websites included a convincing instruction.
The sensible answer is not to ban intermediate reasoning or to treat every model output as dangerous. It is to keep a clear boundary between reasoning and authority.
Human organisations do this already, though not always well. A working note can inform a decision, but a payment needs a signature. A briefing can suggest a course of action, but a director gives the approval. A record can be useful without becoming self-authenticating.
AI systems need the same discipline. The more power they have, the less wise it is to rely on a single textual cue as proof that a command is legitimate.
A useful correction to the word “non-deterministic”
It is tempting to say that the danger exists because AI is non-deterministic. That is only part of the story.
A system whose output can vary from run to run plainly deserves caution. Yet even a perfectly repeatable model would still face the provenance problem if it was given ambiguous or adversarial text. It might make the same wrong interpretation every time.
The deeper difficulty is that language is a poor medium for enforcing authority by itself. The model sees labels, instructions, quotations, examples, partial plans and external documents as text in a shared context. It can often infer the intended hierarchy. “Often” is not a security guarantee.
That is why the paper’s experiment matters. It gives us a way to examine a failure mode that is otherwise too easy to describe vaguely. The authors did not prove that every model will mistake fake reasoning for its own process. They showed that, in the models and settings they tested, reasoning-like style could create a measurable and consequential form of role confusion.
That finding should change how we think about the boundary between a helpful assistant and an autonomous agent with real permissions.
A model can be excellent at drafting, comparing, summarising and proposing. It can be valuable precisely because its routes to an answer are sometimes inventive. But when the task involves money, private data, public publication or irreversible system changes, invention is not the same thing as authority.
The model may have produced a persuasive working note. The next question remains the one a careful person should always ask.
Who wrote it, and what is it allowed to authorise?
Further reading: Charles Ye, Jasmine Cui and Dylan Hadfield-Menell, “Prompt Injection as Role Confusion” (ICML 2026, arXiv v6, 27 June 2026), and their project write-up. The paper reports the CoT Forgery result, including the approximately 60% attack-success figure in its experimental setting; that figure is the authors’ measurement, not a general rate for every model or AI product.